Network element allocation method and device and storage medium

Abstract

The invention relates to the technical field of communication, and discloses a network element allocation method and device and a storage medium, and the network element allocation method comprises the steps: obtaining candidate network elements meeting a preset condition in response to a received network element allocation service request; when the number of the candidate network elements is greater than 1, obtaining the actual working states of the candidate network elements, constructing a candidate network element state tensor, and constructing a task vector according to the network element allocation service request; and based on the candidate network element state tensor and the task vector, obtaining an optimal network element allocation method through a pre-trained agent so as to select an execution network element from the candidate network elements to process a network element allocation service request. The task allocation strategy is adjusted in real time through the reinforcement learning algorithm, the decision is dynamically optimized according to the current network state and historical data, and the load conditions of a plurality of network elements are comprehensively considered, so that all resources can be reasonably allocated, the future task allocation strategy is optimized, and the task allocation efficiency is improved. And the utilization rate and the service quality of the whole network resource are greatly improved.

Description

Technical Field

[0001] The present invention relates to the field of communication technology, and in particular to a network element allocation method, device, equipment and storage medium. Background Art

[0002] In an IMS (IP Multimedia Subsystem) network, traditional task allocation methods usually rely on predefined rules and policies. The core of these methods is to achieve reasonable task allocation by setting priorities and weights of network elements.

[0003] However, traditional priority and weight-based allocation methods lack dynamic adaptability and flexibility, and are difficult to adapt to dynamically changing network traffic patterns. These methods are often based on simple thresholds to trigger load balancing mechanisms, making it difficult to accurately control load distribution. Moreover, traditional algorithms rely on regular manual evaluation and manual adjustments and lack the ability to self-optimize, which further increases the complexity and cost of operations and maintenance.

[0004] Therefore, there is an urgent need for a network element task allocation method with dynamic adaptability, intelligent decision-making, long-term self-optimization, high flexibility and high degree of automation. Summary of the invention

[0005] In view of this, the present application provides a network element allocation method, device and storage medium, which effectively avoids many shortcomings of traditional task allocation algorithms and provides a more advanced and efficient solution for network element allocation. The technical solution is as follows.

[0006] In a first aspect, the present invention provides a network element allocation method, the method comprising:

[0007] In response to the received network element allocation service request, a network element meeting a preset condition is acquired to obtain a candidate network element.

[0008] When the number of the candidate network elements is greater than 1, the actual working status of the candidate network elements is obtained and a candidate network element status tensor is constructed, and a task vector is constructed according to the network element allocation service request.

[0009] Based on the candidate network element state tensor and the task vector, an optimal network element allocation method is obtained through a pre-trained intelligent agent to select an execution network element from the candidate network elements to process the network element allocation service request.

[0010] The network element allocation method provided by the present invention has the following advantages:

[0011] The network element allocation method of the present invention starts the network element allocation process after the IP multimedia subsystem receives the network element allocation service request of the user. The system queries the operation status information of all current network elements and selects qualified candidate network elements from them according to specific rules. The candidate network elements selected here need to support the service type required by the user's network element allocation service request. When the number of selected candidate network elements is greater than 1, it is necessary to make further decisions through the intelligent agent. The decision generation of the intelligent agent needs to comprehensively consider the current status of each candidate network element and the necessary information of the current network element allocation service request. The intelligent agent will combine the actual working status of the candidate network element and the network element allocation service request, analyze the current status through the deep reinforcement learning algorithm, and select the most suitable network element as the execution network element of the task request to process the user's network element allocation service request, that is, to obtain the optimal network element allocation decision. The decision of the intelligent agent is based on the experience accumulated during its training process to achieve the optimal task allocation effect. Subsequently, the IP multimedia subsystem converts the decision of the intelligent agent into an operation instruction for executing the network element to process the network element allocation service request based on the mapping relationship between the potential actions of the intelligent agent and the network element. The network element allocation method of the present invention continuously interacts with the environment through a reinforcement learning algorithm, adjusts the task allocation strategy in real time, and dynamically optimizes decisions based on the current network status and historical data, thereby ensuring that the optimal network element task allocation can be achieved under any circumstances. It also comprehensively considers the load conditions of multiple network elements to achieve more refined load balancing, which not only avoids the overload problem of a single network element, but also ensures that all resources can be reasonably allocated, optimizes the strategy for future task allocation, and thus greatly improves the overall network resource utilization and service quality.

[0012] In an optional embodiment, selecting an execution network element from the candidate network elements to process the network element allocation service request includes: according to a mapping relationship, converting the potential actions of the intelligent agent indicated by the optimal network element allocation method into operation instructions of the execution network element to process the network element allocation service request; the mapping relationship is obtained by associating and mapping the candidate network elements and the potential actions of the intelligent agent.

[0013] Specifically, the network elements are associated and mapped with the potential actions of the agent. This means that the state information of each network element will correspond to the potential actions of the agent, so that the potential actions of the agent indicated by the decision of the agent can be reversely mapped back to the actual network element, that is, the decision of the agent is converted into a specific network element operation instruction, so as to actually execute the user's network element allocation service request.

[0014] In an optional implementation, the network element allocation method further includes: when the number of the candidate network elements is 1, using the candidate network element as an execution network element to process the network element allocation service request; when the number of the candidate network elements is 0, starting an exception handling process; the exception handling process includes reporting an error or starting a failover mechanism.

[0015] Specifically, if no NEs meet the conditions after screening, that is, the number of NEs after screening is 0, the system will enter the abnormal process processing stage. This may mean that an error needs to be reported or other failover mechanisms need to be initiated to handle this situation to ensure that the service request is not ignored. If only one NE meets the conditions after screening, the NE allocation service request is directly provided by the NE without the need for the agent to make a decision.

[0016] In an optional implementation manner, the network element that meets the preset condition includes a network element that supports the service type of the network element allocation service request.

[0017] Specifically, according to specific rules, the IP multimedia subsystem selects qualified network elements from the current state set of all network elements. These rules include whether the network element supports the service type required by the network element allocation service request. Only those network elements that can meet these conditions will be considered as candidates. This process ensures that only suitable network elements will be further considered.

[0018] In an optional implementation, the network element allocation method also includes a training process of the network element allocation method, including setting a maximum training step number; when the current step number is not greater than the maximum training step number and the number of training network elements is greater than 1, iteratively training the agent; the training network element is a network element that meets preset conditions. The training process specifically includes: in response to a received network element allocation service request, obtaining a network element that meets preset conditions to obtain a training network element; when the number of training network elements is greater than 1, generating a random number within a specified probability range, and generating a random exploration probability based on the current step length and the maximum training step length; generating an agent execution strategy based on the relationship between the random number and the random exploration probability; selecting a sample network element to process the network element allocation service request based on the agent execution strategy; calculating a reward based on the status of each network element including the sample network element; and updating the network parameters of the agent based on the reward.

[0019] In an optional embodiment, the training process also includes: when the number of the training network elements is 0, starting an exception handling process; the exception handling process includes reporting an error or starting a failover mechanism; when the number of the training network elements is 1, using the training network element as a sample network element to process the network element allocation service request; calculating rewards based on the status of each network element including the sample network element; and updating the network parameters of the intelligent agent based on the rewards.

[0020] In an optional embodiment, the training process generates an agent execution strategy based on the size relationship between the random number and the random exploration probability, including: if the random number is smaller than the random exploration probability, then generating an agent to execute a random strategy; otherwise, generating an agent to execute a preset strategy.

[0021] In a second aspect, the present invention provides a network element allocation device, the device comprising:

[0022] A network element screening module, configured to obtain network elements that meet preset conditions in response to a received network element allocation service request, and obtain candidate network elements;

[0023] An acquisition module, used for acquiring the actual working status of the candidate network element and constructing a candidate network element status tensor when the number of the candidate network element is greater than 1, and constructing a task vector according to the network element allocation service request;

[0024] A decision generation module is used to obtain an optimal network element allocation method through a pre-trained intelligent agent based on the candidate network element state tensor and the task vector to select an execution network element from the candidate network elements to process the network element allocation service request.

[0025] In a third aspect, the present invention provides a computer device, comprising: a memory and a processor, the memory and the processor being communicatively connected to each other, the memory storing computer instructions, and the processor executing the network element allocation method of the first aspect or any corresponding embodiment thereof by executing the computer instructions.

[0026] In a fourth aspect, the present invention provides a computer-readable storage medium having computer instructions stored thereon, the computer instructions being used to enable a computer to execute the network element allocation method of the first aspect or any corresponding embodiment thereof.

[0027] In a fifth aspect, the present invention provides a computer program product, comprising computer instructions, wherein the computer instructions are used to enable a computer to execute the network element allocation method of the first aspect or any corresponding embodiment thereof. BRIEF DESCRIPTION OF THE DRAWINGS

[0028] In order to more clearly illustrate the specific implementation methods of the present invention or the technical solutions in the prior art, the drawings required for use in the specific implementation methods or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some implementation methods of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0029] Figure 1 It is a flow chart of a network element allocation method provided by an embodiment of the present invention;

[0030] Figure 2 is a flowchart of a network element allocation method provided by an embodiment of the present invention;

[0031] Figure 3 It is a training flow diagram of a network element allocation method provided by an embodiment of the present invention;

[0032] Figure 4 It is a training flow chart of the network element allocation method provided by an embodiment of the present invention;

[0033] Figure 5 It is a schematic diagram of a specific training scenario of the network element allocation method provided by an embodiment of the present invention;

[0034] Figure 6 It is a schematic diagram of a specific application scenario of the network element allocation method provided by an embodiment of the present invention;

[0035] Figure 7 It is a structural block diagram of a network element allocation device provided by an embodiment of the present invention;

[0036] Figure 8 It is a schematic diagram of the hardware structure of a computer device according to an embodiment of the present invention. DETAILED DESCRIPTION

[0037] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without making creative work are within the scope of protection of this application.

[0038] It should be understood that the "indication" mentioned in the embodiments of the present application can be a direct indication, an indirect indication, or an indication of an association relationship. For example, A indicates B, which can mean that A directly indicates B, for example, B can be obtained through A; it can also mean that A indirectly indicates B, for example, A indicates C, and B can be obtained through C; it can also mean that there is an association relationship between A and B.

[0039] In the description of the embodiments of the present application, the term "corresponding" may indicate a direct or indirect correspondence between two items, or an association relationship between the two items, or a relationship between indication and being indicated, configuration and being configured, and the like.

[0040] In an embodiment of the present application, "predefinition" can be achieved by pre-saving corresponding codes, tables or other methods that can be used to indicate relevant information in a device (for example, including a terminal device and a network device). The present application does not limit its specific implementation method.

[0041] First, the terms involved in this application are introduced.

[0042] ARQ: Automatic Repeat reQuest, automatic retransmission request; 5G: 5th Generation Mobile Networks, fifth generation mobile communication technology; EBA: Exponential Backoff Algorithm, exponential backoff algorithm; DDPG: Exponential Backoff Algorithm, deep deterministic policy gradient algorithm; AI: Artificial Intelligence, artificial intelligence; IMS: IP Multimedia Subsystem, IP Multimedia Subsystem; I-CSCF and S-CSCF are the core components in the IMS network, I-CSCF: Interrogating-CSCF, interrogating session control function; S-CSCF: Serving-CSCF, serving session control function.

[0043] IMS (IP Multimedia Subsystem) network, or IP Multimedia Subsystem network, is a form of multimedia service based on IP network. In IMS (IP Multimedia Subsystem) network, traditional task allocation methods usually rely on predefined rules and policies. Specifically, in actual operation, when a task arrives, the system will first check the priority of all available network elements. If there are network elements with higher priorities, they will be given priority to be assigned to these network elements to perform tasks. If multiple network elements have the same priority, tasks will be allocated in proportion to their weights. Network elements with higher weights will take on more tasks in proportion to their weights, thereby ensuring the effective use of resources.

[0044] Although the active-standby disaster recovery mechanism of traditional methods has achieved effective load balancing and fault prevention to a certain extent, its limitations are gradually revealed in the face of increasingly complex network environments. Traditional task allocation methods are mainly based on static rules and preset strategies, which are difficult to adapt to dynamically changing network traffic patterns, and are insufficient in dealing with sudden traffic and complex service requirements, and cannot be flexibly adjusted to meet real-time changing needs.

[0045] With the advancement of artificial intelligence technology, especially the integration of deep learning and reinforcement learning, deep reinforcement learning (DRL) as an emerging technical means can self-learn and optimize decision-making strategies through continuous interaction with the environment, and better adapt to dynamic changes in the network. By continuously optimizing the task allocation mechanism, deep reinforcement learning technology can enable the IMS network to better support diversified and high-bandwidth service requirements and provide users with a more stable and smooth communication experience. Especially during burst traffic peaks, DRL can quickly adjust the task allocation strategy to ensure service quality while avoiding network congestion.

[0046] Therefore, the embodiment of the present invention provides a network element allocation method, which adjusts the task allocation strategy in real time through a reinforcement learning algorithm, dynamically optimizes the decision according to the current network status and historical data, and comprehensively considers the load conditions of multiple network elements, ensures that all resources can be reasonably allocated, optimizes the strategy of future task allocation, and greatly improves the overall network resource utilization and service quality. Figure 1 Flowchart of network element allocation method and Figure 2 The flow chart of the network element allocation method is as follows. The flow chart of the network element allocation method includes the following steps.

[0047] S101. In response to a received network element allocation service request, obtain network elements that meet preset conditions to obtain candidate network elements.

[0048] Optionally, in step S101, when the IMS receives the user's network element allocation service request, the startup process is triggered, indicating that the network element allocation task is officially started. At this time, the IMS will query the status set of all current candidate network elements. The purpose of this step is to collect the current operating status information of all potential candidate network elements, including but not limited to indicators such as CPU utilization, memory usage, and task queue length. Then, qualified network elements are screened out from the candidate network elements according to specific rules. These rules include whether the network element supports the service type required by the request, etc. Only network elements that meet specific rules will be selected as candidate network elements, and these candidate network elements will be further screened in subsequent steps.

[0049] S102: When the number of the candidate network elements is greater than 1, the actual working status of the candidate network elements is obtained and a candidate network element status tensor is constructed, and a task vector is constructed according to the network element allocation service request.

[0050] Optionally, in step S102, the next step of the process is selected according to the number of candidate network elements.

[0051] Specifically, if Figure 2 As shown, when the number of candidate network elements is 0, the system will enter the abnormal process processing stage, and it is necessary to report an error or start other failover mechanisms to handle this situation to ensure that the service request will not be ignored.

[0052] When the number of candidate network elements is 1, only one network element meets the condition, and then the network element that meets the condition is used as the execution network element to process the network element allocation service request.

[0053] When the number of candidate network elements is greater than 1, the agent needs to make further decisions to select execution network elements from the candidate network elements to execute the user's network element allocation service request. Before the agent makes a decision, the system also needs to obtain the actual working status of these candidate network elements, including key performance indicators such as CPU occupancy and task queue length. Based on the actual working status information of the network element, a network element status tensor is constructed. This tensor contains detailed information about the current status of each candidate network element, providing an intuitive data representation for the agent. In addition, the system also needs to extract relevant information from the service request and construct a task vector. This vector contains necessary information about the request itself, such as request type, priority, etc. These are important reference factors for the agent to make decisions.

[0054] S103. Based on the candidate network element state tensor and the task vector, obtain an optimal network element allocation method through a pre-trained intelligent agent to select an execution network element from the candidate network elements to process the network element allocation service request.

[0055] Optionally, in step S103, the intelligent agent derives the optimal intelligent agent execution strategy, i.e., the optimal network element allocation method, based on the candidate network element state tensor and task vector, and then converts the potential actions of the intelligent agent indicated by the intelligent agent execution strategy into operation instructions for executing the network element based on the mapping relationship between the network element and the potential actions of the intelligent agent. The operation instructions are used to process the user's network element allocation service request.

[0056] In summary, the network element allocation method provided by the embodiment of the present invention starts the network element allocation process after the IP multimedia subsystem receives the network element allocation service request of the user. The system queries the operation status information of all current network elements and selects qualified candidate network elements from them according to specific rules. The selected candidate network elements need to support the service type required by the user's network element allocation service request. When the number of selected candidate network elements is greater than 1, it is necessary to make further decisions through the intelligent agent. The decision generation of the intelligent agent needs to comprehensively consider the current status of each candidate network element and the necessary information of the current network element allocation service request. The intelligent agent will combine the actual working status of the candidate network element and the network element allocation service request, analyze the current status through the deep reinforcement learning algorithm, and select the most suitable network element as the execution network element of the task request to process the user's network element allocation service request, that is, to obtain the optimal network element allocation decision. The decision of the intelligent agent is based on the experience accumulated during its training process to achieve the optimal task allocation effect. Subsequently, the IP multimedia subsystem converts the decision of the intelligent agent into an operation instruction for executing the network element to process the network element allocation service request based on the mapping relationship between the potential actions of the intelligent agent and the network element. The network element allocation method of the present invention continuously interacts with the environment through a reinforcement learning algorithm, adjusts the task allocation strategy in real time, and dynamically optimizes decisions based on the current network status and historical data, thereby ensuring that the optimal network element task allocation can be achieved under any circumstances. It also comprehensively considers the load conditions of multiple network elements to achieve more refined load balancing, which not only avoids the overload problem of a single network element, but also ensures that all resources can be reasonably allocated, optimizes the strategy for future task allocation, and thus greatly improves the overall network resource utilization and service quality.

[0057] In an optional embodiment, the network element allocation method provided in the above embodiment also includes a training process of the network element allocation method, and the iterative logic of the training process is: setting the maximum number of training steps; when the current number of steps is not greater than the maximum number of training steps and the number of qualified training network elements is greater than 1, iterative training is performed on the intelligent agent.

[0058] Combination Figure 3 Training flow chart of network element allocation method and Figure 4 A training process flow chart of a network element allocation method is provided, wherein the training process specifically includes the following steps.

[0059] S301. In response to a received network element allocation service request, obtain a network element that meets a preset condition and obtain a training network element.

[0060] The preset conditions in step 301 are the same as those in step S101 of the above embodiment, and are not described again here.

[0061] S302: When the number of training network elements is greater than 1, a random number is generated within a specified probability range, and a random exploration probability is generated according to the current step size and the maximum training step size.

[0062] Optionally, in step S302, if Figure 4 As shown in the figure, when the number of training network elements is greater than 1, it means that the agent needs to perform the next step of screening. The agent will select the execution strategy based on the relationship between the random number and the random exploration probability, generate a random number p, which is a random number seed between 0 and 1, and use the random number seed as the simulated agent exploration probability. Then, the random exploration probability is obtained based on the current training step size and the maximum training step size, and the simulated exploration probability is compared with the random exploration probability. The expression of the random exploration probability is:

[0063] f(x)=1-e k-episode ;

[0064] Where k is the current training step size and episode is the maximum training step size.

[0065] S303: Generate an agent execution strategy based on the relationship between the random number and the random exploration probability.

[0066] Optionally, in step S303, if the random number is less than the random exploration probability, an agent is generated to execute a random strategy; if the random number is greater than or equal to the random exploration probability, an agent is generated to execute a preset strategy.

[0067] S304: Select a sample network element according to the agent execution strategy to process the network element allocation service request.

[0068] Specifically, in step 304, based on the mapping relationship between network elements and agent potential actions, the agent potential actions indicated by the agent execution strategy are converted into operation instructions for executing the network elements, and the operation instructions are used to process the user's network element allocation service request.

[0069] S305: Calculate rewards according to the status of each network element including the sample network element.

[0070] Specifically, in step S305, after the agent generates the execution strategy, the system will obtain the status of the sample network element and the status of all network elements, and give rewards to the agent based on these statuses. The rewards will motivate the agent to select the strategy with the most balanced load among the network elements.

[0071] S306. Update the network parameters of the agent according to the reward.

[0072] Specifically, in step S306, the network parameters of the agent are updated according to the reward obtained. This step is the key in the agent learning process and aims to optimize its decision-making ability. By adjusting the parameters, the agent can gradually improve its strategy to make better decisions in similar situations in the future. Including the relationship between the policy network parameters and the value network parameters.

[0073] Specifically, the value network is updated through the TD algorithm (Temporal Difference), the policy network gradient is calculated through the gradient ascent algorithm, and the policy network parameters and value network parameters are updated through the value network and policy network gradients. The value network scores the current action selected by the agent in the current state, and the policy network is the neural network ultimately used for the agent's decision-making. Its input is the current environment state, and its output is the best action. The specific calculation process is as follows.

[0074] The expression of the TD algorithm updating the value network is:

[0075] Q target =r+γQ(s k+1 ,π(s k+1 |θ)|θ)

[0076] L(θ)=[Q(s k ,a k |θ)-Q target ] 2

[0077] The expression for calculating the policy network gradient by the gradient ascent algorithm is:

[0078]

[0079] Update network parameters:

[0080]

[0081] In the formula, θ and θ are the policy network and value network parameters respectively; α is the learning rate; k is the current training step size; r is the reward value; Q target is the target value, which is used to update the target value of Q value; L(θ) is the value network loss function; J(θ) is the policy network loss function; Q(s,a) is the value network, the input is the state and the policy, and the output is the score of the policy; π(s) is the policy network, the input is the state, and the output is the policy; S k is the state transition result after the kth decision; A k This is the k-th decision.

[0082] In an optional embodiment, if Figure 4As shown, the training process also includes: if the number of training network elements is 0, the exception handling process is started to report the error or start the failover mechanism; if the number of training network elements is 1, the current training network element is used as a sample network element to process the network element allocation service request; the reward is calculated according to the status of each network element including the sample network element; and the network parameters of the intelligent body are updated according to the reward.

[0083] Based on the above embodiment, a specific example is provided below to illustrate the network element allocation method of this embodiment.

[0084] First, the agent needs to be trained. The schematic diagram of the reinforcement learning agent training environment design using the I-CSCF domain selection S-CSCF as an example is as follows: Figure 5 The environment mainly consists of three parts: simulated user system, IMS core network (IMS-Core) and intelligent agent (Agent). The detailed functions and interaction methods of these three parts are as follows.

[0085] The simulated user system continuously initiates requests to the IMS network based on the given initial parameters, including Invite or Register (service or registration) requests, to simulate the high concurrency in real operator scenarios. This part is designed to create a realistic network load environment so that the agent can be trained under conditions close to actual usage. By simulating a large number of concurrent requests, the system can test and verify the performance of the agent under high load conditions.

[0086] When the agent receives the request, it will select the best selected network element, that is, the best S-CSCF, according to the preset decision process. This process involves querying the status information of all current network elements, filtering out qualified network elements according to preset rules, and finally making a decision based on the learning model of the agent. The agent's decision is based on its understanding of the current network status and its prediction of the future status, in order to achieve the best task allocation effect.

[0087] As the core component of the network, IMS-Core is responsible for executing the agent's strategy. When the agent makes a choice, IMS-Core will assign tasks to the corresponding S-CSCF according to the selected strategy. While executing the strategy, IMS-Core will also forward all the status information of the candidate network elements and the selected network elements to the reinforcement learning strategy evaluation center. This information includes but is not limited to key performance indicators such as CPU utilization, memory usage, and task queue length.

[0088] The reinforcement learning strategy evaluation center receives the network element status information forwarded by IMS-Core, and evaluates the behavior of the intelligent agent based on the historical trend of the network element load and the current network element load balancing status. This process involves calculating the impact of the intelligent agent's decision and giving the intelligent agent corresponding reward or penalty signals. The reward mechanism is the core of reinforcement learning, which is used to guide the learning direction of the intelligent agent and help it make better decisions in similar situations in the future.

[0089] The agent is trained through the above training environment so that the agent can generate the best execution decision according to the user's network element allocation service request. After the agent training is completed, the network element allocation method can be applied to a specific scenario. Figure 6 As shown, the selection of S-CSCF in the I-CSCF domain is still taken as an example.

[0090] In this scenario, when the I-CSCF receives an Invite or Register request from a user, it needs to select an appropriate S-CSCF to process the request. The I-CSCF first initiates a Service Authorization Request (SAR) to the Home Subscriber Server (HSS) to obtain the capability set information of all available S-CSCFs. After receiving the capability set returned by the HSS, the I-CSCF will filter out the S-CSCFs that meet the required service capability requirements, and further check the link connection status between these S-CSCFs and whether they are currently functioning normally.

[0091] After completing the above steps, the I-CSCF will collect further status information from the S-CSCF that meets the capability set requirements, including but not limited to the task queue length of the network element and the current CPU occupancy. These status information and the relevant data of the request itself are organized into a comprehensive state representation SS, which is used to describe the current situation or condition of the agent in the environment, including a tensor reflecting the current state of each candidate S-CSCF and a state vector composed of the specific situation of the request.

[0092] Subsequently, this state representation SS is submitted to a pre-trained agent, which uses its built-in reinforcement learning model to analyze the current state and make decisions accordingly. The agent evaluates the current state of each S-CSCF and the specific requirements of the request, and selects an S-CSCF network element that is most suitable for serving the current request. This process not only takes into account the immediate network state, but also optimizes the strategy of future task allocation as much as possible through the learning ability of the reinforcement learning algorithm, so as to achieve long-term resource utilization efficiency and service quality improvement.

[0093] In summary, the network element allocation method provided in this example trains the intelligent agent through deep reinforcement learning methods, continuously interacts with the environment for learning, adjusts the task allocation strategy in real time, and dynamically optimizes the decision according to the current network status and historical data, thereby ensuring that the optimal network element task allocation can be achieved under any circumstances. In addition, the load conditions of multiple network elements are comprehensively considered to achieve more refined load balancing, which not only avoids the overload problem of a single network element, but also ensures that all resources can be reasonably allocated, optimizes the strategy for future task allocation, and thus greatly improves the overall network resource utilization and service quality.

[0094] In the embodiments of the present application, a network element allocation device is also provided, which is used to implement the above embodiments and preferred implementation modes, and the descriptions that have been made will not be repeated. As used below, the term "module" can be a combination of software and / or hardware that implements a predetermined function. Although the devices described in the following embodiments are preferably implemented in software, the implementation of hardware, or a combination of software and hardware, is also possible and conceivable.

[0095] The embodiment of the present application provides a network element allocation device, Figure 7 : is a structural diagram of a network element allocation device provided in an embodiment of the present application, the device comprising:

[0096] The network element screening module 701 is used to obtain network elements that meet preset conditions in response to the received network element allocation service request to obtain candidate network elements;

[0097] An acquisition module 702 is used to acquire the actual working state of the candidate network element and construct a candidate network element state tensor when the number of the candidate network element is greater than 1, and construct a task vector according to the network element allocation service request;

[0098] The decision generation module 703 is used to obtain the optimal network element allocation method through a pre-trained intelligent agent based on the candidate network element state tensor and the task vector to select an execution network element from the candidate network elements to process the network element allocation service request.

[0099] In an optional embodiment, the network element allocation device further includes:

[0100] The policy execution module 704 is specifically used for:

[0101] According to the mapping relationship between the candidate network elements and the potential actions of the agent, the potential actions of the agent indicated by the optimal network element allocation method are converted into operation instructions for executing the network element to process the network element allocation service request.

[0102] In an optional embodiment, the acquisition module 702 is further configured to:

[0103] When the number of the candidate network elements is 1, the candidate network element is used as the execution network element to process the network element allocation service request; when the number of the candidate network elements is 0, the exception handling process is started; the exception handling process includes reporting an error or starting a failover mechanism.

[0104] In an optional embodiment, the decision making module 703 is further configured to:

[0105] A maximum number of training steps is set; when the current number of steps is not greater than the maximum number of training steps and the number of training network elements is greater than 1, the agent is iteratively trained; the training network element is a network element that meets preset conditions.

[0106] In an optional embodiment, the decision making module 703 is further configured to:

[0107] When the number of training network elements is greater than 1, a random number is generated within a specified probability range, and a random exploration probability is generated based on the current step size and the maximum training step size; an agent execution strategy is generated based on the relationship between the random number and the random exploration probability; a sample network element is selected according to the agent execution strategy to process the network element allocation service request; a reward is calculated based on the status of each network element including the sample network element; and the network parameters of the agent are updated based on the reward.

[0108] The further functional description of each of the above modules and units is the same as that of the above corresponding embodiments and will not be repeated here.

[0109] The network element allocation device in this embodiment is presented in the form of a functional unit, where the unit refers to an ASIC (Application Specific Integrated Circuit) circuit, a processor and memory that executes one or more software or fixed programs, and / or other devices that can provide the above functions.

[0110] The embodiment of the present invention also provides a computer device having the above Figure 7 The network element allocation device shown.

[0111] See also Figure 8 , Figure 8 is a schematic diagram of the structure of a computer device provided by an optional embodiment of the present invention, such as Figure 8 As shown, the computer device includes: one or more processors 10, a memory 20, and interfaces for connecting various components, including high-speed interfaces and low-speed interfaces. Various components are connected to each other using different buses for communication, and can be installed on a common mainboard or installed in other ways as needed. The processor can process instructions executed in the computer device, including instructions stored in or on the memory to display graphic information in a graphical user interface on an external input / output device (such as a display device coupled to an interface). In some optional embodiments, if necessary, multiple processors and / or multiple buses can be used together with multiple memories and multiple memories. Similarly, multiple computer devices can be connected, and each device provides some necessary operations (for example, as a server array, a group of blade servers, or a multi-processor system). Figure 8 A processor 10 is taken as an example.

[0112] The processor 10 may be a central processing unit, a network processor or a combination thereof. The processor 10 may further include a hardware chip. The hardware chip may be a dedicated integrated circuit, a programmable logic device or a combination thereof. The programmable logic device may be a complex programmable logic device, a field programmable gate array, a general purpose array logic or any combination thereof.

[0113] The memory 20 stores instructions executable by at least one processor 10, so that the at least one processor 10 executes the method shown in the above embodiment.

[0114] The memory 20 may include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function; the data storage area may store data created according to the use of the computer device, etc. In addition, the memory 20 may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some optional embodiments, the memory 20 may optionally include a memory remotely arranged relative to the processor 10, and these remote memories may be connected to the computer device via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0115] The memory 20 may include a volatile memory, such as a random access memory; the memory may also include a non-volatile memory, such as a flash memory, a hard disk or a solid state drive; the memory 20 may also include a combination of the above types of memory.

[0116] The computer device further comprises a communication interface 30 for the computer device to communicate with other devices or a communication network.

[0117] The embodiment of the present invention also provides a computer-readable storage medium. The method according to the embodiment of the present invention can be implemented in hardware, firmware, or can be implemented as a computer code that can be recorded in a storage medium, or can be implemented as a computer code that is originally stored in a remote storage medium or a non-temporary machine-readable storage medium and will be stored in a local storage medium through a network download, so that the method described herein can be stored in such software processing on a storage medium using a general-purpose computer, a dedicated processor, or programmable or dedicated hardware. Among them, the storage medium can be a magnetic disk, an optical disk, a read-only storage memory, a random access memory, a flash memory, a hard disk or a solid-state hard disk, etc.; further, the storage medium can also include a combination of the above types of memories. It can be understood that a computer, a processor, a microprocessor controller, or programmable hardware includes a storage component that can store or receive software or computer code. When the software or computer code is accessed and executed by a computer, a processor, or hardware, the method shown in the above embodiment is implemented.

[0118] A part of the present invention may be applied as a computer program product, such as a computer program instruction, which, when executed by a computer, can call or provide the method and / or technical solution according to the present invention through the operation of the computer. Those skilled in the art should understand that the existence of the computer program instruction in a computer-readable medium includes, but is not limited to, a source file, an executable file, an installation package file, etc., and accordingly, the way in which the computer program instruction is executed by the computer includes, but is not limited to: the computer directly executes the instruction, or the computer compiles the instruction and then executes the corresponding compiled program, or the computer reads and executes the instruction, or the computer reads and installs the instruction and then executes the corresponding installed program. Here, the computer-readable medium may be any available computer-readable storage medium or communication medium accessible to the computer.

[0119] Although the embodiments of the present invention have been described in conjunction with the accompanying drawings, those skilled in the art may make various modifications and variations without departing from the spirit and scope of the present invention, and such modifications and variations are all within the scope defined by the appended claims.

Claims

1. A network element allocation method, characterized in that: include: In response to the received network element allocation service request, obtaining network elements that meet preset conditions to obtain candidate network elements; When the number of the candidate network elements is greater than 1, the actual working state of the candidate network elements is obtained and a candidate network element state tensor is constructed, and a task vector is constructed according to the network element allocation service request; Based on the candidate network element state tensor and the task vector, an optimal network element allocation method is obtained through a pre-trained intelligent agent to select an execution network element from the candidate network elements to process the network element allocation service request.

2. The method according to claim 1, characterized in that: The selecting an execution network element from the candidate network elements to process the network element allocation service request includes: According to the mapping relationship, the potential actions of the intelligent agent indicated by the optimal network element allocation method are converted into operation instructions for executing network elements to process the network element allocation service request; the mapping relationship is obtained by associating and mapping the candidate network elements and the potential actions of the intelligent agent.

3. The method according to claim 1, characterized in that The method further comprises: When the number of the candidate network elements is 1, the candidate network element is used as an execution network element to process the network element allocation service request; When the number of the candidate network elements is 0, an exception handling process is started; the exception handling process includes reporting an error or starting a failover mechanism.

4. The method according to claim 1, characterized in that The network element meeting the preset condition includes a network element supporting the service type of the network element allocation service request.

5. The method according to claim 1, characterized in that The method further comprises: Set the maximum number of training steps; When the current step number is not greater than the maximum training step number and the number of training network elements is greater than 1, the intelligent agent is iteratively trained; the training network element is a network element that meets preset conditions.

6. The method according to any one of claims 1 to 5, characterized in that: The method further comprises: In response to the received network element allocation service request, acquiring a network element that meets a preset condition to obtain a training network element; When the number of training network elements is greater than 1, a random number is generated within a specified probability range, and a random exploration probability is generated according to the current step size and the maximum training step size; Generate an agent execution strategy according to the random number and the magnitude relationship of the random exploration probability; Selecting a sample network element according to the agent execution strategy to process the network element allocation service request; Calculate rewards according to the status of each network element including the sample network element; The network parameters of the agent are updated according to the reward.

7. The method according to claim 6, characterized in that The method further comprises: When the number of the training network elements is 0, an exception handling process is started; the exception handling process includes reporting an error or starting a failover mechanism; When the number of the training network elements is 1, the training network element is used as a sample network element to process the network element allocation service request; a reward is calculated based on the status of each network element including the sample network element; and the network parameters of the intelligent agent are updated based on the reward.

8. The method according to claim 6, characterized in that The step of generating an agent execution strategy according to the random number and the magnitude relationship between the random exploration probability includes: If the random number is less than the random exploration probability, the generated agent executes the random strategy, otherwise the generated agent executes the preset strategy.

9. A network element allocation device, characterized in that: include: A network element screening module, configured to obtain network elements that meet preset conditions in response to a received network element allocation service request, and obtain candidate network elements; An acquisition module, used for acquiring the actual working status of the candidate network element and constructing a candidate network element status tensor when the number of the candidate network element is greater than 1, and constructing a task vector according to the network element allocation service request; A decision generation module is used to obtain an optimal network element allocation method through a pre-trained intelligent agent based on the candidate network element state tensor and the task vector to select an execution network element from the candidate network elements to process the network element allocation service request.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores computer instructions, and the computer instructions are used to enable a computer to execute the network element allocation method according to any one of claims 1 to 7.

Images