International short message multi-operator dynamic switching control system based on reinforcement learning

Abstract

The invention belongs to the technical field of Internet, and discloses an international short message multi-operator dynamic switching control system based on reinforcement learning. Comprising a monitoring acquisition module which is used for acquiring user short message data and operation data of n operators in real time, and the operation data comprises state data, operation data and network congestion information fed back by the operators; according to the method, short message data and operation data are collected, a loading score and a congestion score are obtained through analysis, an operator selection strategy is automatically optimized according to actual feedback information through reinforcement learning and continuous learning in combination with the short message length and the operation data, and the method can better adapt to a complex and changeable environment; more complex and more effective operator switching modes and rules are mined from a large amount of data and past experience, so that the short message sending cost can be optimized, and the reach rate can be improved; the strategy selected by the operator based on reinforcement learning can sense environment changes such as network congestion and the like, the strategy is rapidly adjusted, and timeliness and reliability of short message service are guaranteed.

Description

Technical Field

[0001] The present invention relates to the field of Internet technology, and more specifically, to an international short message multi-operator dynamic switching control system based on reinforcement learning. Background Art

[0002] International SMS is an important way to transmit information. Traditional international SMS sending systems usually adopt static routing strategies, that is, operators are selected according to pre-set rules or priorities. This method cannot be dynamically adjusted according to real-time network conditions, operator performance and other factors, which can easily lead to problems such as delayed SMS sending and high failure rates. Static systems are difficult to cope with complex and changing network environments and business needs, cannot automatically learn and optimize operator selection strategies, and cannot fully utilize the advantages of different operators, resulting in inefficient resource utilization and increased operating costs.

[0003] The Chinese patent application with publication number CN113965898A discloses a dynamic routing and high availability solution based on multiple SMS providers, including a server for storing operator information, SMS information, and SMS template information; an information display page that presents a data connection relationship with the server, and the information display page is used to display operator docking information and SMS sending statistics; an operator configuration module that presents a data connection relationship with the server and the information presentation page, and the module supports inputting SMS operator account-related information, interface docking information, and SMS issuance weights, and stores them in the server; a weight calculation module for calculating the sending probability of matching operators; and a SMS sending statistics report for counting the number of SMS sent by each operator for settlement. The invention selects SMS operators based on preset weights through the SMS system, which saves costs and is safe and reliable.

[0004] Although the above method can meet most scenarios, research and practical application of the above method and existing technology have found that the above method and existing technology have at least the following defects: Relying on pre-set weights to select operators is relatively static and fixed. Once the weights are set, the system will largely distribute SMS sending tasks according to a predetermined ratio, and cannot flexibly adjust according to factors such as real-time SMS sending results and dynamic changes in operator network conditions. Without a deep policy learning mechanism, it is difficult to dig out more complex and effective operator switching modes and rules from a large amount of data and past experience. When faced with sudden situations such as sudden congestion, failure of the operator network, or a sudden increase in SMS sending delays, it is impossible to make the best response immediately.

[0005] In view of this, the present invention proposes a reinforcement learning international SMS multi-operator dynamic switching control system to solve the above problems. Summary of the invention

[0006] In order to overcome the above-mentioned defects of the prior art and to achieve the above-mentioned purpose, the present invention provides the following technical solution: an international SMS multi-operator dynamic switching control system based on reinforcement learning, comprising: Monitoring and collection module: used to collect user SMS data and operation data of n operators in real time. The operation data includes status data, operation data and network congestion information fed back by operators. The first analysis module is used to use the status data as the input of the operator analysis model to obtain the status score; The second analysis module is used to use the network congestion information as the input of the congestion analysis model to obtain the congestion scores of the base stations of the operators corresponding to the longitude and latitude coordinates of the receiving area, that is, the congestion scores of the base stations of each operator near the longitude and latitude coordinates of the receiving area; Intelligent decision-making module: takes the SMS length, the status score, operation data and congestion score of each of the n operators as the input of the reinforcement learning algorithm, obtains the optimal operator, and generates the corresponding operator switching instructions; Dynamic switching module: used for selecting a corresponding operator to send the text message according to the operator switching instruction when executing the sending of the text message corresponding to the user text message data according to the real-time operator switching instruction.

[0007] Furthermore, the reinforcement learning algorithm defines an agent for selecting an operator; the agent selects the optimal operator based on a strategy model; and the training method of the strategy model includes: S1. Mark the operator's SMS sending system as an environment, which includes SMS length, status score, operation data and congestion score. The state vector is obtained by splicing and normalization. , aggregate the operators into the same set, use the set as the action space, and mark the selected operator as an action; set up to store the experience tuple An experience replay buffer of size J; S2, set X training rounds; S3. Reset the initial state vector of each training round to the initial state ; S4, the agent according to the time step The corresponding state vector , through the online network to predict the agent at time step The corresponding Q value ; S5. The agent selects actions through the ε-greedy strategy ; S6, preset the number of steps c per interval, and sample the experience tuple corresponding to s time steps from the experience playback buffer ; For each sampled experience tuple, the Q value of the action selected in the current state is calculated through the online network ; Predict the maximum Q value in the next state after performing the action through the target network ; S7. Calculate the Q value corresponding to the selected action through the target network ; S8. Calculate the loss function based on the Q value calculated by the target network and the online network ; S9, copying the parameters of the online network to the target network according to the preset frequency; S10: When a training round reaches the set maximum number of steps Y or meets the end condition, the training round ends and enters the next training round; S11. Repeat S2-S10 until the agent's strategy converges, and output the corresponding strategy when the strategy converges, that is, the optimal operator.

[0008] Furthermore, the experience tuple The process of interaction between the agent and the environment includes the current time step The state vector , actions taken , the reward value obtained and the state vector at the next time step ; The reward value It is calculated based on the reward function, which is defined as follows: ; ; ; in, Rewards for successfully delivering text messages; The basic reward points for successfully delivering a text message; is the time step The number of successfully delivered SMS messages; For continuous Additional bonus coefficient for sending SMS messages; Penalty for failed SMS delivery; Penalty score for failed SMS sending; is the time step The number of failed SMS messages; is the time step The reward value.

[0009] Furthermore, the online network predicts the agent at time step The corresponding Q value The calculation is as follows:

[0010] Among them, the time step Corresponding is the reward value; is a discount factor, which is used to measure the importance of future rewards to current decisions, and its value range is between 0 and 1; It is the maximum Q value predicted by the online network in the next state after executing the action.

[0011] Furthermore, in S5, -The greedy strategy method of selecting actions includes: S501, dividing the status levels and the corresponding status score intervals, dividing the status levels of the operators according to the status scores, and allocating the selection probability according to the number of operators in different status levels; S502, randomly select layer, based on the reward function, calculate the Layer Each operator in the The Q values ​​within the layer and calculate the probability of selecting the operator with the highest Q value :

[0012] in, For the The operator in Q value of the layer; For the The operator in Q value of the layer; S503, using a random number generator, generates a Random numbers in the interval ,like , then randomly select an operator within the layer, where is the preset initial exploration rate; if , then according to The probability distribution selects the operator; after the agent performs the selected action, the environment corresponding to the time step feedbacks the reward and updates to the next state, and then stores the experience tuple in the experience replay buffer.

[0013] Furthermore, the online network and the target network both belong to a Double DQN network, and the Double DQN network is used to learn and estimate the value of actions under different state vectors. The online network and the target network have the same structure, but different parameters and update frequencies; The training method of the online network and the target network is the same, and the training method includes: Collect W groups of strategy training data in advance, the strategy training data includes strategy data and Q value corresponding to the strategy data; the strategy data includes state vector and congestion score of base station to which the receiving area belongs; Each set of policy training data is used as input of the online network and the target network. The online network and the target network use the Q value corresponding to each set of policy data as output, and use the actual Q value corresponding to each set of policy data as the prediction target; the training target is to minimize the sum of the prediction errors of all Q values; the online network and the target network are trained until the sum of the prediction errors reaches convergence and the training is stopped; the online network and the target network are deep neural network models; the number of input layer nodes of the deep neural network model corresponds to the dimension of the state vector, and the number of output layer nodes corresponds to the size of the action space, that is, the number of operators; The loss function The calculation is as follows:

[0014] in, The parameters of the online network are updated according to the loss function through the back propagation algorithm, so that the online network is optimized in the direction of reducing the prediction error.

[0015] Furthermore, the user SMS data includes the SMS length and the longitude and latitude coordinates of the receiving area; the SMS length is the number of characters contained in the SMS text; the longitude and latitude coordinates of the receiving area are the longitude and latitude coordinates of the center point corresponding to the area where the SMS recipient is located; The status data includes network delay, packet loss rate and traffic transmission volume within a preset time; The operation data is the number of SMS messages successfully delivered by each operator per unit time, the number of SMS messages failed to be sent, and the length of the queue of SMS messages to be sent; The network congestion information is status data obtained when performing network detection on the communication base stations of each operator covering the longitude and latitude coordinates of the receiving area.

[0016] Furthermore, the method for obtaining the network delay and packet loss rate is as follows: Get the service host address of the operator, send h Ping commands to the service host address, transmit q data packets each time, record the round-trip time of each echo request packet feedback and the number of echo request packets received m, calculate the average round-trip time of all echo request packets feedback by h Ping commands as the network delay, and calculate the packet loss rate based on the following formula: ; in, is the packet loss rate; The method for obtaining the traffic transmission amount within the preset time includes: Use a network packet capture tool to capture data packets, and use the statistical function of the network packet capture tool to count the number N of data packets and the size of each data packet within the preset time d, and then obtain the total size S of all data packets. The ratio of the total size S of all data packets to the preset time d is used as the traffic transmission amount.

[0017] Furthermore, the training method of the operator analysis model includes: Collecting group B operator training data in advance, where the operator training data includes state data and state scores corresponding to the state data; Each group of operator training data is used as input of the operator analysis model, the operator analysis model uses the state score corresponding to each group of state data as output, and uses the actual state score corresponding to each group of state data as the prediction target; minimizing the sum of prediction errors of all state scores is used as the training target; the operator analysis model is trained until the sum of prediction errors reaches convergence and the training is stopped; the operator analysis model is a deep neural network model; The operator analysis model loss function value is the mean square error; Prediction error formula: , b is the bth group of state data; B is the number of groups of state data; is the status score corresponding to the status data of group b; is the actual status score corresponding to the status data of group b.

[0018] Furthermore, the training method of the congestion analysis model includes: Collecting a group E of congestion training data in advance, the congestion training data includes network congestion information and congestion scores corresponding to the network congestion information; Each group of congestion training data is used as the input of the congestion analysis model. The congestion analysis model uses the congestion score corresponding to each group of network congestion information as the output, and uses the actual congestion score corresponding to each group of network congestion information as the prediction target; minimizing the sum of the prediction errors of all congestion scores is used as the training target; the congestion analysis model is trained until the sum of the prediction errors reaches convergence, and the training is stopped; the congestion analysis model is a deep neural network model; The congestion analysis model loss function value is the mean square error; Prediction error formula: , e is the e-th group of network congestion information; E is the number of groups of network congestion information; is the congestion score corresponding to the e-th group of network congestion information; is the actual congestion score corresponding to the e-th group of network congestion information.

[0019] The technical effects and advantages of the international SMS multi-operator dynamic switching control system based on reinforcement learning of the present invention are as follows: The present invention uses reinforcement learning, continuous trial and error, and learning to automatically optimize the operator selection strategy based on actual feedback information, and can better adapt to complex and changing environments; and from a large amount of data and past experience, more complex and effective operator switching modes and rules are excavated, which can optimize the cost of SMS sending and improve the reach rate; the operator selection strategy based on reinforcement learning can perceive environmental changes such as network congestion, and quickly adjust the strategy to respond, so as to better ensure the timeliness and reliability of SMS services. BRIEF DESCRIPTION OF THE DRAWINGS

[0020] Figure 1 This is a system block diagram mentioned in Embodiment 1 of the present invention; Figure 2 This is a schematic diagram of the reinforcement learning algorithm architecture mentioned in Example 1 of the present invention; Figure 3 This is a flow chart of the strategy model training mentioned in Example 1 of the present invention; Figure 4 The method mentioned in Example 1 of the present invention -Flowchart of the method for selecting actions with the greedy strategy. DETAILED DESCRIPTION

[0021] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0022] Example 1 See also Figure 1 As shown, this embodiment provides an international SMS multi-operator dynamic switching control system based on reinforcement learning.

[0023] Monitoring and collection module: used to collect user SMS data and operation data of n operators in real time. The operation data includes status data, operation data and network congestion information fed back by operators. The user SMS data is obtained by connecting to the user-side data, including the SMS length and the longitude and latitude coordinates of the receiving area; the SMS length is the number of characters contained in the SMS text; by collecting the SMS length, it is convenient for the subsequent intelligent decision-making module to select the appropriate operator according to the network bandwidth of different operators. For example, for longer SMS, select an operator with wider network bandwidth and less network congestion to ensure that the SMS can be sent smoothly, reduce the interruption or delay caused by insufficient network resources, and thus improve the stability of SMS sending services.

[0024] The longitude and latitude coordinates of the receiving area are the longitude and latitude coordinates of the center point corresponding to the area where the SMS recipient is located; by collecting the longitude and latitude coordinates of the receiving area, it is convenient to obtain network congestion information of the receiving area for subsequent analysis, reduce SMS sending congestion, and improve the probability and stability of successful SMS sending.

[0025] Status data is obtained by testing the operator's network, including network delay, packet loss rate and traffic transmission volume within a preset time; collecting this status data is crucial for selecting SMS operators; low network delay and packet loss rate can reduce SMS sending time and errors, ensuring timeliness and accuracy; traffic transmission volume can help determine the operator's network load capacity; combining these data can accurately select stable and reliable operators, effectively improving the stability of SMS sending services.

[0026] The method to obtain network delay and packet loss rate is as follows: Get the service host address of the operator, send h Ping commands to the service host address, transmit q data packets each time, record the round-trip time of each echo request packet feedback and the number of echo request packets received m, calculate the average round-trip time of all echo request packets feedback by h Ping commands as the network delay, and calculate the packet loss rate based on the following formula: ; in, is the packet loss rate.

[0027] Methods for obtaining the traffic transmission volume within a preset time include: Use a network packet capture tool to capture data packets, and use the statistical function of the network packet capture tool to count the number N of data packets and the size of each data packet within the preset time d, and then obtain the total size S of all data packets. The ratio of the total size S of all data packets to the preset time d is used as the traffic transmission amount.

[0028] Operation data is obtained by counting the operation-related data of each operator within a unit time, specifically the number of SMS messages successfully delivered by each operator within a unit time, the number of SMS messages failed to be sent, and the length of the queue of SMS messages to be sent. Collecting the number of SMS messages successfully delivered by each operator can reflect the delivery capabilities of the operator, which is conducive to ensuring that SMS messages can be effectively delivered; collecting the number of SMS messages failed to be sent can reflect the service reliability of the operator. Collecting the length of the queue of SMS messages to be sent can reflect the load situation. If the queue length is short, the waiting time for SMS messages can be reduced, reducing the risk of sending delays. Taking these data into consideration, operators that can improve the stability of SMS sending can be accurately selected.

[0029] The network congestion information is the status data obtained when performing network detection on the communication base stations of each operator covering the longitude and latitude coordinates of the receiving area. It can reflect the network status of the communication base stations of each operator covering the longitude and latitude coordinates of the receiving area. When the communication base station belonging to a certain operator is congested, it is convenient for the system to select other operators in time, reduce the congestion of SMS sending, and improve the probability and stability of successful SMS sending.

[0030] The first analysis module is used to use the status data as the input of the operator analysis model to obtain the status score.

[0031] The training methods of the operator analysis model include: A group B of operator training data is collected in advance, where the operator training data includes state data and state scores corresponding to the state data.

[0032] Each group of operator training data is used as the input of the operator analysis model. The operator analysis model takes the state score corresponding to each group of state data as the output, and the actual state score corresponding to each group of state data as the prediction target; minimizing the sum of prediction errors of all state scores is used as the training target; the operator analysis model is trained until the sum of prediction errors reaches convergence and the training is stopped; the operator analysis model is a deep neural network model.

[0033] The operator analysis model loss function value is the mean square error.

[0034] Prediction error formula: , b is the bth group of state data; B is the number of groups of state data; is the status score corresponding to the status data of group b; is the actual status score corresponding to the status data of group b.

[0035] The second analysis module is used to use the network congestion information as the input of the congestion analysis model to obtain the congestion score of the base station of the operator corresponding to the longitude and latitude coordinates of the receiving area, that is, the congestion score of the base station of each operator near the longitude and latitude coordinates of the receiving area, taking into account the network conditions of each base station on the receiving side, to avoid congestion of the SMS corresponding to the user's SMS data in the later stage of sending, so as to facilitate the selection of stable and reliable operators in the entire process of sending the SMS corresponding to the user's SMS data, and realize the stability of the SMS sending service throughout the process.

[0036] The training methods for the congestion analysis model include: A group E of congestion training data is collected in advance, and the congestion training data includes network congestion information and congestion scores corresponding to the network congestion information.

[0037] Each group of congestion training data is used as the input of the congestion analysis model. The congestion analysis model uses the congestion score corresponding to each group of network congestion information as the output, and the actual congestion score corresponding to each group of network congestion information as the prediction target; minimizing the sum of the prediction errors of all congestion scores is used as the training target; the congestion analysis model is trained until the sum of the prediction errors reaches convergence, and the training is stopped; the congestion analysis model is a deep neural network model.

[0038] The congestion analysis model loss function value is the mean square error.

[0039] Prediction error formula: , e is the e-th group of network congestion information; E is the number of groups of network congestion information; is the congestion score corresponding to the e-th group of network congestion information; is the actual congestion score corresponding to the e-th group of network congestion information.

[0040] Intelligent decision-making module: The SMS length, the status score of each of the n operators, the operation data and the congestion score are used as the input of the reinforcement learning algorithm to obtain the optimal operator and generate the corresponding operator switching instruction; wherein, the value of n is large enough. When the number of operators is less than n, it is padded by zeros to ensure that the number of operators reaches n.

[0041] See also Figure 2 , the training methods of reinforcement learning algorithms include: Define an agent for selecting operators; the agent selects the best operator based on the policy model to achieve the optimization goal of SMS sending; See also Figure 3 , the training methods of the strategy model include: S1. Model initialization: The operator's SMS sending system is marked as the environment, which includes SMS length, status score, operation data and congestion score. The SMS length, status score, operation data and congestion score are concatenated (such as [SMS length, operator A status score, operator A operation data, operator A congestion score, operator B status score, operator B operation data, operator B congestion score, ...]), and the state vector is obtained after normalization. ,Normalization can ensure that the data value range is in a similar space, which is convenient for the algorithm to converge.,The operators are aggregated into the same set, and the set is used as the action space, and the selected operators are marked as actions; the size of the action space corresponds to the number of operators.

[0042] The operator's SMS sending system performs the action corresponding to the current time step at the operator After that, the time step will be fed back according to the reward function Corresponding reward value And update the state vector corresponding to the next time step .

[0043] Reward Value Based on the reward function calculation, the reward function is defined as follows: ; ; ; in, Rewards for successfully delivering text messages; The basic reward points for successfully delivering a text message; is the time step The number of successfully delivered SMS messages; For continuous Additional bonus coefficient for sending SMS messages; Penalty for failed SMS delivery; Penalty score for failed SMS sending; is the time step The number of failed SMS messages; is the time step The reward value of By setting up such a reward formula, rewards or punishments can be accurately given based on the delivery status of SMS and the number of consecutive successes, thereby guiding operators to focus on improving service reliability and strengthening global coordination awareness.

[0044] Set an experience replay buffer of size J to store the current time step during the interaction between the agent and the environment The state vector at the next time step, the action taken, the reward value obtained, and the experience tuple of the state vector at the next time step .

[0045] S2. Set X training rounds; each training round is a complete process of the agent interacting with the environment and learning.

[0046] S3. At the initial stage of each training round, the state vector is reset to the initial state ; Ensure the consistency and independence of the environment state at the beginning of each training round, without being disturbed by the end state of the previous round.

[0047] S4, the agent according to the time step The corresponding state vector , through the online network to predict the agent at time step The corresponding Q value ; This step is the basis for subsequent action selection and strategy learning, and is used to estimate the value of different actions in the current state through an online network.

[0048] The online network predicts the agent at time step The corresponding Q value The calculation is as follows:

[0049] Among them, the time step Corresponding is the reward value; is a discount factor, which is used to measure the importance of future rewards to current decisions, and its value range is between 0 and 1; It is the maximum Q value predicted by the online network in the next state after executing the action.

[0050] S5. The agent selects actions through the ε-greedy strategy ; The agent performs the selected action After that, the time step Corresponding environmental feedback rewards And update to the next state ; The experience tuple Stored into the experience replay buffer.

[0051] See also Figure 4 ,pass -The greedy strategy method of selecting actions includes: S501, divide the status levels (such as the first level, the second level, and the third level) and the corresponding status score intervals, divide the status levels of the operators according to the status scores, and allocate selection probabilities according to the number of operators in different status levels; if the operators in the first level account for 60% of the total number of operators, the selection probability of the first level is 0.6.

[0052] S502, randomly select layer, based on the reward function, calculate the Layer Each operator in the The Q value within the layer (see the online network prediction agent at time step The corresponding Q value calculation step) and calculate the probability of selecting the operator with the highest Q value :

[0053] in, For the The operator in Q value of the layer; For the The operator in Q value of the layer; S503, using a random number generator, generates a Random numbers in the interval ,like , then randomly select an operator within the layer, where is the preset initial exploration rate; if , then according to The probability distribution selects the operator; after the agent performs the selected action, the environment corresponding to the time step feedbacks the reward and updates to the next state, and then stores the experience tuple in the experience replay buffer.

[0054] S6, preset the number of steps c per interval, and sample the experience tuple corresponding to s time steps from the experience playback buffer ; For each sampled experience tuple, the Q value of the action selected in the current state is calculated through the online network ; Predict the maximum Q value in the next state after performing the action through the target network ; Both the online network and the target network belong to the Double DQN network. The Double DQN network is used to learn and estimate the value of actions under different state vectors. The online network and the target network have the same structure, but different parameters and different update frequencies.

[0055] The training methods of the online network and the target network are the same, and the training methods include: W groups of strategy training data are collected in advance, and the strategy training data includes strategy data and Q values ​​corresponding to the strategy data; the strategy data includes a state vector and a congestion score of a base station to which a receiving area belongs.

[0056] Each set of policy training data is used as the input of the online network and the target network. The online network and the target network use the Q value corresponding to each set of policy data as the output, and the actual Q value corresponding to each set of policy data as the prediction target; the training target is to minimize the sum of the prediction errors of all Q values; the online network and the target network are trained until the sum of the prediction errors reaches convergence, and the training is stopped; the online network and the target network are deep neural network models. The number of input layer nodes of the deep neural network model corresponds to the dimension of the state vector, and the number of output layer nodes corresponds to the size of the action space, that is, the number of operators.

[0057] S7. Calculate the Q value corresponding to the selected action through the target network (See online network prediction of intelligent agents at time step The corresponding Q value calculation steps).

[0058] S8. Calculate the loss function based on the Q value calculated by the target network and the Q value calculated by the online network :

[0059] in, The parameters of the online network are updated according to the loss function through the back propagation algorithm, so that the online network is optimized in the direction of reducing the prediction error.

[0060] S9. According to the preset target network update frequency, the parameters of the online network are copied to the target network, so that the parameters of the target network are updated synchronously with the parameters of the online network to ensure that the target network can provide a relatively stable target Q value reference for the online network, which is conducive to stable training.

[0061] S10. When a training round reaches the set maximum number of steps Y or meets other end conditions (such as the same action selection results for M consecutive times), the training round ends and enters the next training round.

[0062] S11. Repeat the above training cycle process. After multiple rounds of training, the agent's strategy will gradually converge. By continuously interacting with the environment, storing experience, and updating the network, the agent can learn the strategy of selecting the optimal operator in different environmental states, that is, output the optimal operator.

[0063] Dynamic switching module: used for selecting a corresponding operator to send the text message according to the operator switching instruction when executing the sending of the text message corresponding to the user text message data according to the real-time operator switching instruction.

[0064] The above is only a specific embodiment of the present invention, but the protection scope of the present invention is not limited thereto. Any person skilled in the art can easily think of changes or substitutions within the technical scope disclosed by the present invention, which should be included in the protection scope of the present invention. Therefore, the protection scope of the present invention shall be based on the protection scope of the claims.

[0065] Finally: The above description is only a preferred embodiment of the present invention and is not intended to limit the present invention. Any modifications, equivalent substitutions, improvements, etc. made within the spirit and principles of the present invention should be included in the protection scope of the present invention.

Claims

1. An international SMS multi-operator dynamic switching control system based on reinforcement learning, characterized in that: include: Monitoring and collection module: used to collect user SMS data and operation data of n operators in real time. The operation data includes status data, operation data and network congestion information fed back by operators. The first analysis module is used to use the status data as the input of the operator analysis model to obtain the status score; The second analysis module is used to use the network congestion information as the input of the congestion analysis model to obtain the congestion scores of the base stations of the operators corresponding to the longitude and latitude coordinates of the receiving area, that is, the congestion scores of the base stations of each operator near the longitude and latitude coordinates of the receiving area; Intelligent decision-making module: takes the SMS length, the status score, operation data and congestion score of each of the n operators as the input of the reinforcement learning algorithm, obtains the optimal operator, and generates the corresponding operator switching instructions; Dynamic switching module: used for selecting a corresponding operator to send the text message according to the operator switching instruction when executing the sending of the text message corresponding to the user text message data according to the real-time operator switching instruction.

2. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 1 is characterized in that: The reinforcement learning algorithm defines an agent for selecting operators; the agent selects the optimal operator based on a strategy model; the training method of the strategy model includes: S1. Mark the operator's SMS sending system as an environment, which includes SMS length, status score, operation data and congestion score. The state vector is obtained by splicing and normalization. , aggregate the operators into the same set, use the set as the action space, and mark the selected operator as an action; set up to store the experience tuple An experience replay buffer of size J; S2, set X training rounds; S3. Reset the initial state vector of each training round to the initial state ; S4, the agent according to the time step The corresponding state vector , through the online network to predict the agent at time step The corresponding Q value ; S5. The agent selects actions through the ε-greedy strategy ; S6, preset the number of steps c per interval, and sample the experience tuple corresponding to s time steps from the experience playback buffer ; For each sampled experience tuple, the Q value of the action selected in the current state is calculated through the online network ; Predict the maximum Q value in the next state after performing the action through the target network ; S7. Calculate the Q value corresponding to the selected action through the target network ; S8. Calculate the loss function based on the Q value calculated by the target network and the online network ; S9, copying the parameters of the online network to the target network according to the preset frequency; S10: When a training round reaches the set maximum number of steps Y or meets the end condition, the training round ends and enters the next training round; S11. Repeat S2-S10 until the agent's strategy converges, and output the corresponding strategy when the strategy converges, that is, the optimal operator.

3. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 2 is characterized in that: The experience tuple The process of interaction between the agent and the environment includes the current time step The state vector , actions taken , the reward value obtained and the state vector at the next time step ; The reward value It is calculated based on the reward function, which is defined as follows: ； ； ； in, Rewards for successfully delivering text messages; The basic reward points for successfully delivering a text message; is the time step The number of successfully delivered SMS messages; For continuous Additional bonus coefficient for sending SMS messages; Penalty for failed SMS delivery; Penalty score for failed SMS sending; is the time step The number of failed SMS messages; is the time step The reward value.

4. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 2 is characterized in that: The online network predicts the agent at time step The corresponding Q value The calculation is as follows: ； Among them, the time step Corresponding is the reward value; is a discount factor, which is used to measure the importance of future rewards to current decisions, and its value range is between 0 and 1; It is the maximum Q value predicted by the online network in the next state after executing the action.

5. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 2 is characterized in that: In S5, -The greedy strategy method of selecting actions includes: S501, dividing the status levels and the corresponding status score intervals, dividing the status levels of the operators according to the status scores, and allocating the selection probability according to the number of operators in different status levels; S502, randomly select layer, based on the reward function, calculate the Layer Each operator in the The Q values ​​within the layer and calculate the probability of selecting the operator with the highest Q value : ； in, For the The operator in Q value of the layer; For the The operator in Q value of the layer; S503, using a random number generator, generates a Random numbers in the interval ,like , then randomly select an operator within the layer, where is the preset initial exploration rate; if , then according to The probability distribution selects the operator; after the agent performs the selected action, the environment corresponding to the time step feedbacks the reward and updates to the next state, and then stores the experience tuple in the experience replay buffer.

6. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 2 is characterized in that: The online network and the target network both belong to the Double DQN network, which is used to learn and estimate the value of actions under different state vectors. The online network and the target network have the same structure, but different parameters and update frequencies; The online network and the target network are trained in the same way, and the training method includes: Collect W groups of strategy training data in advance, the strategy training data includes strategy data and Q value corresponding to the strategy data; the strategy data includes state vector and congestion score of base station to which the receiving area belongs; Each set of policy training data is used as input of the online network and the target network. The online network and the target network use the Q value corresponding to each set of policy data as output, and use the actual Q value corresponding to each set of policy data as the prediction target; the training target is to minimize the sum of the prediction errors of all Q values; the online network and the target network are trained until the sum of the prediction errors reaches convergence and the training is stopped; the online network and the target network are deep neural network models; the number of input layer nodes of the deep neural network model corresponds to the dimension of the state vector, and the number of output layer nodes corresponds to the size of the action space, that is, the number of operators; The loss function The calculation is as follows: ； in, The parameters of the online network are updated according to the loss function through the back propagation algorithm, so that the online network is optimized in the direction of reducing the prediction error.

7. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 1 is characterized in that: The user SMS data includes the SMS length and the longitude and latitude coordinates of the receiving area; the SMS length is the number of characters contained in the SMS text; the longitude and latitude coordinates of the receiving area are the longitude and latitude coordinates of the center point corresponding to the area where the SMS recipient is located; The status data includes network delay, packet loss rate and traffic transmission volume within a preset time; The operation data is the number of SMS messages successfully delivered by each operator per unit time, the number of SMS messages failed to be sent, and the length of the queue of SMS messages to be sent; The network congestion information is status data obtained when performing network detection on the communication base stations of each operator covering the longitude and latitude coordinates of the receiving area.

8. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 7 is characterized in that: The method for obtaining the network delay and packet loss rate is as follows: Get the service host address of the operator, send h Ping commands to the service host address, transmit q data packets each time, record the round-trip time of each echo request packet feedback and the number of echo request packets received m, calculate the average round-trip time of all echo request packets feedback by h Ping commands as the network delay, and calculate the packet loss rate based on the following formula: ； in, is the packet loss rate; The method for obtaining the traffic transmission amount within the preset time includes: Use a network packet capture tool to capture data packets, and use the statistical function of the network packet capture tool to count the number N of data packets and the size of each data packet within the preset time d, and then obtain the total size S of all data packets. The ratio of the total size S of all data packets to the preset time d is used as the traffic transmission amount.

9. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 1 is characterized in that: The training method of the operator analysis model includes: Collecting group B operator training data in advance, where the operator training data includes state data and state scores corresponding to the state data; Each group of operator training data is used as input of the operator analysis model, the operator analysis model uses the state score corresponding to each group of state data as output, and uses the actual state score corresponding to each group of state data as the prediction target; minimizing the sum of prediction errors of all state scores is used as the training target; the operator analysis model is trained until the sum of prediction errors reaches convergence and the training is stopped; the operator analysis model is a deep neural network model; The operator analysis model loss function value is the mean square error; Prediction error formula: , b is the bth group of state data; B is the number of groups of state data; is the status score corresponding to the status data of group b; is the actual status score corresponding to the status data of group b.

10. The international SMS multi-operator dynamic switching control system based on reinforcement learning according to claim 1, characterized in that: The training method of the congestion analysis model includes: Collecting a group E of congestion training data in advance, the congestion training data includes network congestion information and congestion scores corresponding to the network congestion information; Each group of congestion training data is used as the input of the congestion analysis model. The congestion analysis model uses the congestion score corresponding to each group of network congestion information as the output, and uses the actual congestion score corresponding to each group of network congestion information as the prediction target; minimizing the sum of the prediction errors of all congestion scores is used as the training target; the congestion analysis model is trained until the sum of the prediction errors reaches convergence, and the training is stopped; the congestion analysis model is a deep neural network model; The congestion analysis model loss function value is the mean square error; Prediction error formula: , e is the e-th group of network congestion information; E is the number of groups of network congestion information; is the congestion score corresponding to the e-th group of network congestion information; is the actual congestion score corresponding to the e-th group of network congestion information.

Images