Dynamic channel configuration system and method for multi-region short message service

Abstract

The invention discloses a dynamic channel configuration system and method for multi-region short message service, and relates to the technical field of information transmission, and the method comprises the steps: for each channel, obtaining the average time delay, the short message sending success rate and the average cost of successfully sending 10000 test short messages, calculating a price advantage coefficient, and calculating the comprehensive quality index of each channel; defining a first-layer action space and a second-layer action space based on the comprehensive quality index, defining a state s as a service attribute feature vector of a short message sending task, defining an action a of an intelligent agent, and defining a first-layer return function and a second-layer return function; a hierarchical reinforcement learning model is adopted, a channel cluster selection value function is defined, a channel cluster selection strategy objective function is designed, a channel selection Q value function is defined, a channel selection strategy objective function is designed, and the hierarchical reinforcement learning model is optimized; a channel is distributed for a real-time short message sending task based on a hierarchical reinforcement learning model, and dynamic channel configuration of multi-region short message service is realized.

Description

Technical Field

[0001] The present application relates to the field of information transmission technology, and in particular to a dynamic channel configuration system and method for multi-regional SMS services. Background Art

[0002] In order to meet the needs of SMS services in multiple regions, SMS service providers usually connect SMS channels in multiple regions and dynamically select the best channel to send SMS based on the location of the target mobile phone number.

[0003] The patent application with publication number CN111246406A discloses a method, system, storage medium and terminal device for sending short messages. The method includes obtaining interface documents corresponding to each short message channel, and configuring channel information corresponding to each short message channel according to each interface document, wherein the channel information includes the request method corresponding to the short message channel and the correspondence between the channel field in the short message channel and the system source field; when receiving a short message sending request from a user, determining the target short message channel from the short message channel according to the short message sending request, and determining the target user and source field content corresponding to the short message sending request; converting the system source field into the target channel field corresponding to the target short message channel according to the correspondence, and determining the source field content corresponding to the system source field as the target field content corresponding to the target channel field; obtaining the target request method corresponding to the target short message channel, and sending the short message to the target user using the target request method and the target field content.

[0004] Traditional methods usually select channels based on the effect of a single delivery. This one-time decision-making approach lacks consideration of long-term benefits and does not take into account the impact that the current channel selection may have on future delivery effects. Summary of the invention

[0005] The present application aims to solve at least one of the technical problems in the prior art to a certain extent. To this end, one purpose of the present application is to propose a dynamic channel configuration system and method for multi-regional SMS services, thereby realizing dynamic channel configuration for multi-regional SMS services.

[0006] One aspect of the present application provides a dynamic channel configuration method for a multi-region SMS service, including:

[0007] Step S100: for each channel, obtain its average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages, calculate the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculate the comprehensive quality index of each channel;

[0008] Step S200: Based on the comprehensive quality index, define the first-layer action space and the second-layer action space of the agent, define the state s as the business attribute feature vector of the SMS sending task, define the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, and define the first-layer reward function and the second-layer reward function;

[0009] Step S300: adopting a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defining a channel cluster selection value function and designing a channel cluster selection strategy objective function, defining a channel selection Q value function and designing a channel selection strategy objective function, and optimizing the hierarchical reinforcement learning model;

[0010] Step S400: obtaining a real-time SMS sending task, distributing channels for the real-time SMS sending task based on a hierarchical reinforcement learning model, and sending the task;

[0011] For each channel, the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages are obtained, and the price advantage coefficient between the unit price of the channel and the average unit price of other channels is calculated. The specific method for calculating the comprehensive quality index of each channel is as follows:

[0012] Step S110: Randomly send on channel ai Test SMS messages, record the time difference between sending and receiving each test SMS, get the delay sample set, calculate the arithmetic mean of the delay sample set, and get the average delay of channel ai ;

[0013] Step S120: Take the maximum value of the average delay of all channels as the delay normalization denominator ;

[0014] Step S130: Preset time window , within the time window ct, the total number of test SMS messages sent on channel ai is counted and recorded as , the number of test SMS messages sent successfully is , calculate the ratio between the number of successfully sent test SMS and the total number of sent test SMS, and get the SMS sending success rate of channel ai ;

[0015] Step S140: Obtaining the accessibility adjustment coefficient of the area where the channel ai is located ;

[0016] Step S150: Obtain the unit price from the price list of channel ai, denoted as , calculate the average cost of channel ai for each successful sending of 10,000 test SMS messages ;

[0017] Step S160: Obtain the average unit price of other channels in the same region as channel ai , calculate the average unit price of other channels and the unit price of each channel Price advantage coefficient ;

[0018] Step S170: For each channel, define a comprehensive quality index function, and calculate the comprehensive quality index of each channel ai based on the calculated average delay, delay normalization denominator, SMS sending success rate, reachability adjustment coefficient, average cost and price advantage coefficient ;

[0019] The specific method of defining the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defining the state s as the business attribute feature vector of the SMS sending task, and defining the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s is as follows:

[0020] Step S210: All channels are combined into a channel set, and the i-th channel in the channel set The comprehensive quality index is ;

[0021] Step S220: taking the comprehensive quality index of each channel as a data point, randomly selecting the comprehensive quality indexes of M channels as initial M cluster centers;

[0022] Step S230: For each channel , calculate the Euclidean distance between it and the M cluster centers, assign each channel to the channel cluster with the cluster center closest to it, and for each channel cluster, update its cluster center to the mean of the comprehensive quality index of all channels in the channel cluster when the channel allocation is completed, and repeatedly assign each channel to the channel cluster with the updated cluster center until all the cluster centers and the data points in the channel clusters where the cluster centers are located no longer change, and obtain K channel clusters; K = M;

[0023] Step S240: define two layers of action space. The first layer of action space represents the selection of channel clusters. Defined as the first-level action space ,in is the Kth channel cluster; the second-level action space represents the channel cluster Select a channel ,in, represents the kth channel cluster The Jth channel in; j≤J; k≤K;

[0024] Step S250: define the state s as a service attribute feature vector of the SMS sending task, wherein the service attribute feature vector includes the amount of SMS sent Target area , Sending time window ;

[0025] Step S260: Define action a as the agent selecting a channel cluster from the first layer action space in state s , and from the channel cluster Select a channel Execute SMS sending;

[0026] Step S270: Define a hierarchical action selection strategy for the agent to select action a in state s ,in, Indicates the selection of channel cluster in state s The probability of Indicates that in state s, from the channel cluster Select channel The probability of

[0027] The calculation method of the first-layer reward function is:

[0028] Step S280: For the first layer action space, define the first layer reward function as , get the channel cluster The average comprehensive quality index of all channels in , get the channel cluster used in state s Average cost , based on channel clusters The mean of the average delay of the internal channels and the mean of the SMS sending success rate are used to calculate the channel cluster selected under state s. Synergy , select the channel cluster under calculation state s In the target area Matching Rewards , based on the average comprehensive quality index, average cost, synergy effect, and matching reward, calculate the first-layer reward function, and define the cumulative reward of the first-layer action space based on the first-layer reward function ;

[0029] The calculation method of the second-layer reward function is:

[0030] Step S290: For the second-layer action space, define the second-layer reward function as , construct the current state-action triplet according to the current state, the selected channel cluster and the channel, construct the historical state-action triplet sequence according to the state of the historical N time steps, the selected channel cluster and the channel, and calculate the similarity between the current state-action triplet and the historical state-action triplet sequence , get the channel continuity reward, calculate the channel selection under state s In the target area Resource usage penalty , based on channel clusters Middle Channel Comprehensive quality indicators , use channels in status s Average cost , select channel cluster in state s Synergy , channel continuity reward and resource occupation penalty, calculate the second-layer reward function, and define the cumulative reward of the second-layer action space based on the second-layer reward function ;

[0031] The hierarchical reinforcement learning model is adopted, and the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and the specific method for optimizing the hierarchical reinforcement learning model is:

[0032] Step S310: adopting a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, and defining a channel cluster selection value function of the hierarchical reinforcement learning model , indicating the selection of channel cluster in state s The long-term value of the channel cluster selection strategy is designed based on the channel cluster selection value function. ;

[0033] Step S320: Define channel selection Q value function , which means that in state s, from the channel cluster Select channel The long-term value of the channel selection strategy is designed based on the channel selection Q value function;

[0034] Step S330: construct the loss function of the hierarchical reinforcement learning model, use the stochastic gradient descent algorithm to minimize the loss function, use the policy gradient method to maximize the channel cluster selection strategy objective function, and update the parameters of the channel cluster selection strategy , use the policy gradient method to maximize the channel selection strategy objective function and update the parameters of the channel selection strategy , and obtain the optimized hierarchical reinforcement learning model;

[0035] The specific method of obtaining the real-time SMS sending task, distributing the real-time SMS sending task channels based on the hierarchical reinforcement learning model, and sending the task is as follows:

[0036] Step S410: acquiring a real-time SMS sending task, and extracting a service attribute feature vector of the real-time SMS sending task;

[0037] Step S420: the service attribute feature vector of the real-time SMS sending task is input as state s into the hierarchical reinforcement learning model, the first-layer action space is decided by the channel cluster selection strategy, the probability of selecting each channel cluster under state s is calculated, and the channel cluster selection strategy outputs the channel cluster selected by the first-layer action space;

[0038] Step S430: input the state s and the selected channel cluster into the channel selection strategy, calculate the probability of selecting each channel under the state s and the selected channel cluster, make a decision on the second-layer action space through the channel selection strategy, and output the channel selected by the second-layer action space;

[0039] Step S440: Distribute the real-time SMS sending task to the selected channel for sending.

[0040] One aspect of the present application provides a dynamic channel configuration system for multi-regional SMS services, including:

[0041] The channel quality calculation module obtains the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages for each channel, calculates the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculates the comprehensive quality index of each channel;

[0042] The agent definition module defines the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defines the state s as the business attribute feature vector of the SMS sending task, defines the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, and defines the first-layer reward function and the second-layer reward function;

[0043] A reinforcement learning optimization module adopts a hierarchical reinforcement learning model, wherein the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and optimizes the hierarchical reinforcement learning model;

[0044] The real-time SMS sending module obtains the real-time SMS sending task, distributes the channels for the real-time SMS sending task based on the hierarchical reinforcement learning model, and sends the task.

[0045] One aspect of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps in the dynamic channel configuration method for multi-regional SMS services are implemented.

[0046] One aspect of the present application provides a readable storage medium storing a computer program, wherein the computer program is suitable for loading by a processor to execute the steps in a dynamic channel configuration method for multi-regional SMS services.

[0047] The dynamic channel configuration system and method for multi-regional SMS services proposed in this application have the following advantages over the prior art:

[0048] The calculation of the comprehensive quality indicators of this application fully considers key factors such as channel latency, success rate, cost, as well as external influences such as regional differences and market competition, making the evaluation results closer to actual business needs.

[0049] This application upgrades the original channel-level action space to the channel cluster level, greatly reducing the dimension of the action space and simplifying the exploration difficulty of the reinforcement learning algorithm. Channels in the same cluster have similar performance, and selecting channels in the same cluster has little effect on the sending effect. This hierarchical action space design is more in line with the characteristics of actual tasks.

[0050] In the first and second-layer reward functions defined in this application, not only the comprehensive quality of the channel and the sending cost are taken into consideration, but also multiple influencing factors such as synergy, matching, continuity, etc. are considered for rewarding or punishing, guiding the reinforcement learning algorithm to weigh various influencing factors and balance short-term and long-term effects.

[0051] This application uses a hierarchical reinforcement learning model to model and optimize the channel selection process, defines a channel cluster selection value function and a channel selection Q value function, and models the original problem as a hierarchical Markov decision process that first makes decisions at the cluster level and then at the channel level. Channel cluster selection considers long-term returns, and channel selection considers local optimality. The entire model takes into account both long-term benefits and local optimality, overcoming the short-sightedness of single decisions in traditional methods.

[0052] This application applies the trained hierarchical reinforcement learning model to the actual SMS sending task, dynamically adjusts the channel combination according to the specific needs of each task, realizes personalized optimization of task granularity, improves the delivery rate and timeliness of SMS, and enhances user satisfaction. BRIEF DESCRIPTION OF THE DRAWINGS

[0053] Figure 1 A method flow chart of a dynamic channel configuration method for a multi-region SMS service provided by this application;

[0054] Figure 2 A flow chart of the calculation method of the comprehensive quality index provided for this application;

[0055] Figure 3 A flowchart of the construction process of the hierarchical reinforcement learning model provided for this application;

[0056] Figure 4 Functional module diagram of the dynamic channel configuration system for multi-region SMS services provided in this application. DETAILED DESCRIPTION

[0057] In order to better understand the present application, various aspects of the present application will be described in more detail with reference to the accompanying drawings. It should be understood that these detailed descriptions are only descriptions of exemplary embodiments of the present application and do not limit the scope of the present application in any way. Throughout the specification, the same reference numerals refer to the same elements. The expression "and / or" includes any and all combinations of one or more of the associated listed items.

[0058] In the accompanying drawings, the size, dimensions and shapes of the elements have been slightly adjusted for ease of illustration. The drawings are for illustration only and are not strictly drawn to scale. As used herein, the terms "substantially", "approximately" and similar terms are used as terms of approximation, not as terms of degree, and are intended to illustrate the inherent deviations in measurements or calculations that will be recognized by those of ordinary skill in the art. In addition, in this application, the order in which the steps are described does not necessarily represent the order in which these processes occur in actual operation, unless otherwise specified or can be derived from the context.

[0059] It should also be understood that expressions such as "including", "comprising", "having", "containing" and / or "comprising" are open rather than closed expressions in this specification, which indicate the presence of the stated features, elements and / or components, but do not exclude the presence of one or more other features, elements, components and / or combinations thereof. In addition, when expressions such as "at least one of..." appear after a list of listed features, they modify the entire list of features rather than just the individual elements in the list. In addition, when describing embodiments of the present application, "may" is used to mean "one or more embodiments of the present application". And, the term "exemplary" is intended to refer to an example or illustration.

[0060] Unless otherwise defined, all words (including engineering terms and scientific and technological terms) used in this article have the same meaning as those commonly understood by ordinary technicians in the field to which this application belongs. It should also be understood that unless there is a clear explanation in this application, words defined in commonly used dictionaries should be interpreted as having the same meaning as their meaning in the context of the relevant technology, and should not be interpreted in an idealized or overly formal sense.

[0061] It should be noted that, in the absence of conflict, the embodiments and features of the embodiments in the present application can be combined with each other. The present application will be described in detail below with reference to the accompanying drawings and in combination with the embodiments.

[0062] Example 1

[0063] like Figure 1 As shown, the dynamic channel configuration method for multi-region SMS service provided by this application includes:

[0064] Step S100: for each channel, obtain its average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages, calculate the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculate the comprehensive quality index of each channel;

[0065] For each channel, the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages are obtained, and the price advantage coefficient between the unit price of the channel and the average unit price of other channels is calculated. The specific method for calculating the comprehensive quality index of each channel is as follows:

[0066] Step S110: Randomly send on channel ai Test SMS messages, record the time difference between sending and receiving each test SMS, get the delay sample set, calculate the arithmetic mean of the delay sample set, and get the average delay of channel ai ;

[0067] Step S120: Take the maximum value of the average delay of all channels as the delay normalization denominator ;

[0068] Step S130: Preset time window , within the time window ct, the total number of test SMS messages sent on channel ai is counted and recorded as , the number of test SMS messages sent successfully is , calculate the ratio between the number of successfully sent test SMS and the total number of sent test SMS, and get the SMS sending success rate of channel ai ;

[0069] The calculation formula for the SMS sending success rate of the channel ai is: ;

[0070] Step S140: Obtain the accessibility adjustment coefficient of the area where channel ai is located based on expert experience scoring ;

[0071] The accessibility adjustment coefficient is used to reflect the influence of factors such as the maturity of the regional communication infrastructure and control measures on the channel success rate. According to the communication infrastructure conditions, control measures and other factors in the region where the channel ai is located, technicians in this field set the accessibility adjustment coefficient of the region where the channel ai is located based on experience, and the interval is [0, 1];

[0072] Step S150: Obtain the unit price from the price list of channel ai, denoted as , calculate the average cost of channel ai for each successful sending of 10,000 test SMS messages ;

[0073] The calculation formula for the average cost of the channel ai for each successful sending of 10,000 test SMS messages is: ;

[0074] Step S160: Obtain the average unit price of other channels in the same region as channel ai , calculate the average unit price of other channels and the unit price of each channel Price advantage coefficient ;

[0075] The calculation formula of the price advantage coefficient is: ;

[0076] Step S170: For each channel, define a comprehensive quality index function, and calculate the comprehensive quality index of each channel ai based on the calculated average delay, delay normalization denominator, SMS sending success rate, reachability adjustment coefficient, average cost and price advantage coefficient ;

[0077] The functional expression of the comprehensive quality index is: , where α is the delay penalty coefficient, is a smoothing term, β is the success rate reward coefficient, and γ is the cost-effectiveness elasticity coefficient;

[0078] The values ​​of the delay penalty coefficient, smoothing term, success rate reward coefficient, and cost-effectiveness elasticity coefficient are set by those skilled in the art based on experience, wherein the delay penalty coefficient, success rate reward coefficient, and cost-effectiveness elasticity coefficient are all greater than 1, and the smoothing term is a small positive number close to 0, which is used to avoid the abnormal situation where the denominator is 0;

[0079] The comprehensive quality indicators include three indicators: latency, success rate, and cost-effectiveness;

[0080] like Figure 2 As shown, it is a flow chart of the calculation method of the comprehensive quality index provided in this application;

[0081] The above step S100 can comprehensively and objectively evaluate the comprehensive quality of each channel by obtaining various performance indicators of the channel and introducing the accessibility adjustment coefficient and the price advantage coefficient, thereby providing a reliable data basis for subsequent channel clustering and intelligent selection.

[0082] Step S200: Based on the comprehensive quality index, define the first-layer action space and the second-layer action space of the agent, define the state s as the business attribute feature vector of the SMS sending task, define the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, and define the first-layer reward function and the second-layer reward function;

[0083] The specific method of defining the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defining the state s as the business attribute feature vector of the SMS sending task, and defining the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s is as follows:

[0084] Step S210: All channels are combined into a channel set, and the i-th channel in the channel set The comprehensive quality index is ;

[0085] Step S220: taking the comprehensive quality index of each channel as a data point, randomly selecting the comprehensive quality indexes of M channels as initial M cluster centers;

[0086] M is the number of clusters, and the value of M is set by those skilled in the art based on experience;

[0087] Step S230: For each channel , calculate the Euclidean distance between it and the M cluster centers, assign each channel to the channel cluster with the cluster center closest to it, and for each channel cluster, update its cluster center to the mean of the comprehensive quality index of all channels in the channel cluster when the channel allocation is completed, and repeatedly assign each channel to the channel cluster with the updated cluster center until all the cluster centers and the data points in the channel clusters where the cluster centers are located no longer change, and obtain K channel clusters; K = M;

[0088] Step S240: define two layers of action space. The first layer of action space represents the selection of channel clusters. Defined as the first-level action space ,in is the Kth channel cluster; the second-level action space represents the channel cluster Select a channel ,in, represents the kth channel cluster The Jth channel in; j≤J; k≤K;

[0089] Step S250: define the state s as a service attribute feature vector of the SMS sending task, wherein the service attribute feature vector includes the amount of SMS sent Target area , Sending time window ;

[0090] Step S260: Define action a as the agent selecting a channel cluster from the first layer action space in state s , and from the channel cluster Select a channel Execute SMS sending;

[0091] Step S270: Define a hierarchical action selection strategy for the agent to select action a in state s ,in, Indicates the selection of channel cluster in state s The probability of Indicates that in state s, from the channel cluster Select channel probability;

[0092] The calculation method of the first-layer reward function is:

[0093] Step S280: For the first layer action space, define the first layer reward function as , get the channel cluster The average comprehensive quality index of all channels in , get the channel cluster used in state s Average cost , based on channel clusters The mean of the average delay of the internal channels and the mean of the SMS sending success rate are used to calculate the channel cluster selected under state s. Synergy , select the channel cluster under calculation state s In the target area Matching Rewards , based on the average comprehensive quality index, average cost, synergy effect, and matching reward, calculate the first-layer reward function, and define the cumulative reward of the first-layer action space based on the first-layer reward function ;

[0094] Select channel cluster under the state s The calculation formula of the synergistic effect is: , where J represents the channel cluster Total number of channels, Represents channel cluster The average delay of the jth channel in Represents channel cluster The success rate of SMS sending in the jth channel, , are weight coefficients of average delay and SMS sending success rate, respectively, which are set by technicians in this field based on experience;

[0095] Select channel cluster under the state s In the target area The calculation formula for the matching reward is: ,in, Represents channel cluster Historical performance rating in target regions, Represents channel cluster Relevance to the target region;

[0096] The channel cluster The calculation method of historical performance score in the target area is: Statistical historical time channel cluster Total number of SMS messages sent within the channel, statistics channel cluster The number of SMS messages sent successfully within a period of time is calculated by calculating the ratio of the number of SMS messages sent successfully to the total number of SMS messages sent to obtain the channel cluster. SMS delivery success rate in the target area , calculate channel clusters The average of the average delays of all channels in the target area is obtained to obtain the channel cluster Average latency in the target area , count the maximum SMS sending success rate of all channel clusters in different target areas , calculate the minimum average delay of all channel clusters , calculate channel clusters Historical performance ratings in target regions;

[0097] The channel cluster The historical performance score in the target region is calculated as: ;

[0098] The channel cluster The formula for calculating the relevance to the target region is: ,in, Represents channel cluster Coverage in target areas, represents the maximum coverage among all channel clusters;

[0099] The function expression of the first-layer reward function is: ,in, is the weight coefficient of the synergistic effect, which is set by those skilled in the art based on experience;

[0100] The calculation formula for the cumulative return of the first-level action space is: , where t represents the time step and Y represents the discount factor. represents the state at time step t, represents the channel cluster selected at time step t, Represents the state-action sequence of channel cluster selection in an SMS sending task;

[0101] The calculation method of the second-layer reward function is:

[0102] Step S290: For the second-layer action space, define the second-layer reward function as , construct the current state-action triplet according to the current state, the selected channel cluster and the channel, construct the historical state-action triplet sequence according to the state of the historical N time steps, the selected channel cluster and the channel, and calculate the similarity between the current state-action triplet and the historical state-action triplet sequence , get the channel continuity reward, calculate the channel selection under state s In the target area Resource usage penalty , based on channel clusters Middle Channel Comprehensive quality indicators , use channels in status s Average cost , select channel cluster in state s Synergy , channel continuity reward and resource occupation penalty, calculate the second-layer reward function, and define the cumulative reward of the second-layer action space based on the second-layer reward function ;

[0103] The calculation formula of the resource occupation penalty is: ,in, Representation channel The resource utilization rate in the target area is calculated by using the actual communication resource utilization of the channel in the target area. is the penalty coefficient of resource occupancy rate, which is set by technical personnel in this field based on experience;

[0104] The channel The calculation formula for resource occupancy in the target area is: ,in, Representation channel The amount of resources currently used in the target area, Indicates the total communication resources available in the target area;

[0105] The calculation formula of the second-layer reward function is: ,in, is the weight coefficient of similarity;

[0106] The similarity between the current state-action triplet and the historical state-action triplet sequence is the cosine similarity;

[0107] The calculation formula for the cumulative return of the second-level action space is: ,in, The channels selected for time step t;

[0108] The goal of the agent is to maximize the cumulative reward of the first action space and the cumulative reward of the second action space of the SMS sending task;

[0109] like Figure 3 As shown, it is a flow chart of the construction process of the hierarchical reinforcement learning model provided in this application;

[0110] The above step S200 uses a clustering method to group channels to form multiple channel clusters, which can take into account both local optimization and global optimization when selecting channels, ensuring the complementarity of channels within the same cluster and performing balanced scheduling between clusters, thereby improving the overall performance of the system and providing a basis for the establishment of the subsequent hierarchical reinforcement learning model.

[0111] Step S300: adopting a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defining a channel cluster selection value function and designing a channel cluster selection strategy objective function, defining a channel selection Q value function and designing a channel selection strategy objective function, and optimizing the hierarchical reinforcement learning model;

[0112] The hierarchical reinforcement learning model is adopted, and the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and the specific method for optimizing the hierarchical reinforcement learning model is:

[0113] Step S310: adopting a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, and defining a channel cluster selection value function of the hierarchical reinforcement learning model , indicating the selection of channel cluster in state s The long-term value of the channel cluster selection strategy is designed based on the channel cluster selection value function. ;

[0114] The channel cluster selection value function The function expression is: ,in, Indicates the starting state, Indicates in status Next, select the channel cluster; Represents the expected cumulative reward obtained under a given state and action;

[0115] The functional expression of the objective function of the channel cluster selection strategy is: ,in, represents the parameters of the channel cluster selection strategy, D represents the state distribution, and the objective function of the channel cluster selection strategy is expressed on the state distribution D. the expected cumulative return generated;

[0116] Step S320: Define channel selection Q value function , which means that in state s, from the channel cluster Select channel The long-term value of the channel selection strategy is designed based on the channel selection Q value function;

[0117] The function expression of the channel selection Q value function is: ,in, Indicates status Under Channel Cluster The channel selected in

[0118] The functional expression of the channel selection strategy objective function is: ,in, represents the channel cluster selection strategy, Select the parameters of the strategy for the channel;

[0119] Step S330: construct the loss function of the hierarchical reinforcement learning model, use the stochastic gradient descent algorithm to minimize the loss function, use the policy gradient method to maximize the channel cluster selection strategy objective function, and update the parameters of the channel cluster selection strategy , use the policy gradient method to maximize the channel selection strategy objective function and update the parameters of the channel selection strategy , and obtain the optimized hierarchical reinforcement learning model;

[0120] The loss function The calculation formula is: ;

[0121] The above step S300 uses a hierarchical reinforcement learning model to model and optimize the channel selection process, which can autonomously learn and extract experience from massive data, continuously improve the channel cluster selection strategy and channel selection strategy, and make the system adaptive and robust.

[0122] Step S400: obtaining a real-time SMS sending task, distributing channels for the real-time SMS sending task based on a hierarchical reinforcement learning model, and sending the task;

[0123] The specific method of obtaining the real-time SMS sending task, distributing the real-time SMS sending task channels based on the hierarchical reinforcement learning model, and sending the task is as follows:

[0124] Step S410: acquiring a real-time SMS sending task, and extracting a service attribute feature vector of the real-time SMS sending task;

[0125] Step S420: the service attribute feature vector of the real-time SMS sending task is input as state s into the hierarchical reinforcement learning model, the first-layer action space is decided by the channel cluster selection strategy, the probability of selecting each channel cluster under state s is calculated, and the channel cluster selection strategy outputs the channel cluster selected by the first-layer action space;

[0126] Step S430: input the state s and the selected channel cluster into the channel selection strategy, calculate the probability of selecting each channel under the state s and the selected channel cluster, make a decision on the second-layer action space through the channel selection strategy, and output the channel selected by the second-layer action space;

[0127] Step S440: Distribute the real-time SMS sending task to the selected channel for sending.

[0128] The above step S400 applies the trained hierarchical reinforcement learning model to the actual SMS sending task, which can automatically complete the intelligent selection and distribution of channels, greatly improve the work efficiency of the system, and reduce the need for manual intervention.

[0129] Example 2

[0130] like Figure 4 As shown, the dynamic channel configuration system for multi-region SMS services provided by this application includes:

[0131] The channel quality calculation module obtains the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages for each channel, calculates the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculates the comprehensive quality index of each channel;

[0132] The agent definition module defines the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defines the state s as the business attribute feature vector of the SMS sending task, defines the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, and defines the first-layer reward function and the second-layer reward function;

[0133] A reinforcement learning optimization module adopts a hierarchical reinforcement learning model, wherein the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and optimizes the hierarchical reinforcement learning model;

[0134] The real-time SMS sending module obtains the real-time SMS sending task, distributes the channels for the real-time SMS sending task based on the hierarchical reinforcement learning model, and sends the task.

[0135] Example 3

[0136] The present application also provides an electronic device. The electronic device may include one or more processors and one or more memories. The memories store computer-readable codes, and when the computer-readable codes are executed by the one or more processors, the dynamic channel configuration method for the multi-region SMS service may be executed.

[0137] The method or system according to the embodiment of the present application can also be implemented with the help of the architecture of the electronic device disclosed in the present application. The electronic device may include a bus, one or more CPUs, a read-only memory (ROM), a random access memory (RAM), a communication port connected to a network, an input / output component, a hard disk, etc. A storage device in the electronic device, such as a ROM or a hard disk, can store the dynamic channel configuration method for the multi-regional SMS service provided in the present application. The dynamic channel configuration method for multi-regional SMS service may include, for example: for each channel, obtaining its average delay, SMS sending success rate, and average cost of each successfully sent 10,000 test SMS, calculating the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculating the comprehensive quality index of each channel; defining the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defining the state s as the business attribute feature vector of the SMS sending task, defining the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, defining the first-layer reward function and the second-layer reward function; using a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defining a channel cluster selection value function and designing a channel cluster selection strategy objective function, defining a channel selection Q value function and designing a channel selection strategy objective function, and optimizing the hierarchical reinforcement learning model; obtaining a real-time SMS sending task, distributing channels for the real-time SMS sending task based on the hierarchical reinforcement learning model, and sending. Further, the electronic device may also include a user interface.

[0138] Example 4

[0139] The present application also discloses a readable storage medium. The readable storage medium stores computer-readable instructions. When the computer-readable instructions are executed by a processor, the dynamic channel configuration method for a multi-regional SMS service disclosed in the present application can be executed. The readable storage medium includes, but is not limited to, volatile memory and / or non-volatile memory. Volatile memory may include, for example, random access memory (RAM) and cache memory. Non-volatile memory may include, for example, read-only memory (ROM), hard disk, flash memory, etc.

[0140] In addition, according to the implementation mode of the present application, the process disclosed in the present application can be implemented as a computer software program. For example, the present application provides a non-transitory machine-readable storage medium, which stores machine-readable instructions, and the machine-readable instructions can be executed by a processor to execute instructions corresponding to the method steps provided in the present application, for example: for each channel, obtain its average delay, SMS sending success rate, and the average cost of each successful sending of 10,000 test SMS messages, calculate the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculate the comprehensive quality index of each channel; define the first-level action space and the second-level action space of the intelligent agent based on the comprehensive quality index, and define the state s as the SMS sending. The business attribute feature vector of the task, define the action a of the intelligent agent as selecting a channel cluster from the first layer action space and selecting a channel from the second layer action space under state s, define the first layer reward function and the second layer reward function; adopt a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, define a channel cluster selection value function and design a channel cluster selection strategy objective function, define a channel selection Q value function and design a channel selection strategy objective function, optimize the hierarchical reinforcement learning model; obtain a real-time SMS sending task, distribute channels for the real-time SMS sending task based on the hierarchical reinforcement learning model, and send it. When the computer program is executed by a central processing unit (CPU), the above functions defined in the method of the present application are executed.

[0141] The methods, apparatuses, and devices of the present application may be implemented in many ways. For example, the methods, apparatuses, and devices of the present application may be implemented by software, hardware, firmware, or any combination of software, hardware, and firmware. The above order of steps for the method is for illustration only, and the steps of the method of the present application are not limited to the order specifically described above unless otherwise specified. In addition, in some embodiments, the present application may also be implemented as programs recorded in a recording medium, which include machine-readable instructions for implementing the method according to the present application. Therefore, the present application also covers recording media storing programs for executing the method according to the present application.

[0142] In addition, the parts of the above-mentioned technical solutions provided in the embodiments of the present application that are consistent with the implementation principles of the corresponding technical solutions in the prior art are not described in detail to avoid excessive redundancy.

[0143] The specific implementation modes as described above further describe the purpose, technical solutions and beneficial effects of the present invention in detail. It should be understood that the above description is only a specific implementation mode of the present invention and is not intended to limit the present invention. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present invention shall be included in the protection scope of the present invention.

Claims

1. A dynamic channel configuration method for a multi-region SMS service, characterized in that: include: For each channel, obtain its average latency, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages, calculate the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculate the comprehensive quality index of each channel; Based on the comprehensive quality index, the first-layer action space and the second-layer action space of the agent are defined. The state s is defined as the business attribute feature vector of the SMS sending task. The action a of the agent is defined as selecting a channel cluster from the first-layer action space and a channel from the second-layer action space under the state s. The first-layer reward function and the second-layer reward function are defined. Adopting a hierarchical reinforcement learning model, the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defining a channel cluster selection value function and designing a channel cluster selection strategy objective function, defining a channel selection Q value function and designing a channel selection strategy objective function, and optimizing the hierarchical reinforcement learning model; Obtain real-time SMS sending tasks, distribute channels for real-time SMS sending tasks based on the hierarchical reinforcement learning model, and send them.

2. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 1, characterized in that: For each channel, the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages are obtained, and the price advantage coefficient between the unit price of the channel and the average unit price of other channels is calculated. The specific method for calculating the comprehensive quality index of each channel is as follows: Randomly send on channel ai Test SMS messages, record the time difference between sending and receiving each test SMS, get the delay sample set, calculate the arithmetic mean of the delay sample set, and get the average delay of channel ai ; Take the maximum value of the average delay of all channels as the delay normalization denominator ; Preset time window , within the time window ct, the total number of test SMS messages sent on channel ai is counted and recorded as , the number of test SMS messages sent successfully is , calculate the ratio between the number of successfully sent test SMS and the total number of sent test SMS, and get the SMS sending success rate of channel ai ; Get the accessibility adjustment coefficient of the region where the channel ai is located ; Get the unit price from the price list of channel ai, denoted as , calculate the average cost of channel ai for each successful sending of 10,000 test SMS messages ; Get the average unit price of other channels in the same region as channel ai , calculate the average unit price of other channels and the unit price of each channel Price advantage coefficient ; For each channel, define a comprehensive quality index function, and calculate the comprehensive quality index of each channel AI based on the calculated average delay, delay normalization denominator, SMS sending success rate, reachability adjustment coefficient, average cost and price advantage coefficient. .

3. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 2, characterized in that: The specific method of defining the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defining the state s as the business attribute feature vector of the SMS sending task, and defining the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s is as follows: All channels form a channel set, and the i-th channel in the channel set The comprehensive quality index is ; Taking the comprehensive quality index of each channel as the data point, randomly select the comprehensive quality indexes of M channels as the initial M cluster centers; For each channel , calculate the Euclidean distance between it and the M cluster centers, assign each channel to the channel cluster with the cluster center closest to it, and for each channel cluster, update its cluster center to the mean of the comprehensive quality index of all channels in the channel cluster when the channel allocation is completed, and repeatedly assign each channel to the channel cluster with the updated cluster center until all the cluster centers and the data points in the channel clusters where the cluster centers are located no longer change, and obtain K channel clusters; K = M; Define two layers of action space. The first layer of action space represents the selection of channel clusters. Defined as the first-level action space ,in is the Kth channel cluster; the second-level action space represents the channel cluster Select a channel ,in, represents the kth channel cluster The Jth channel in; j≤J; k≤K; Define state s as the business attribute feature vector of the SMS sending task, which includes the amount of SMS sent Target area , Sending time window ; Action a is defined as the agent selecting a channel cluster from the first-layer action space in state s. , and from the channel cluster Select a channel Execute SMS sending; Define a hierarchical action selection strategy for the agent to select action a in state s ,in, Indicates the selection of channel cluster in state s The probability of Indicates that in state s, from the channel cluster Select channel probability.

4. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 3, characterized in that: The calculation method of the first-layer reward function is: For the first-layer action space, the first-layer reward function is defined as , get the channel cluster The average comprehensive quality index of all channels in , get the channel cluster used in state s Average cost , based on channel clusters The mean of the average delay of the internal channels and the mean of the SMS sending success rate are used to calculate the channel cluster selected under state s. Synergy , select the channel cluster under calculation state s In the target area Matching Rewards , based on the average comprehensive quality index, average cost, synergy effect, and matching reward, calculate the first-layer reward function, and define the cumulative reward of the first-layer action space based on the first-layer reward function .

5. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 4, characterized in that: The calculation method of the second-layer reward function is: For the second-layer action space, the second-layer reward function is defined as , construct the current state-action triplet according to the current state, the selected channel cluster and the channel, construct the historical state-action triplet sequence according to the state of the historical N time steps, the selected channel cluster and the channel, and calculate the similarity between the current state-action triplet and the historical state-action triplet sequence , get the channel continuity reward, calculate the channel selection under state s In the target area Resource usage penalty , based on channel clusters Middle Channel Comprehensive quality indicators , use channels in status s Average cost , select channel cluster in state s Synergy , channel continuity reward and resource occupation penalty, calculate the second-layer reward function, and define the cumulative reward of the second-layer action space based on the second-layer reward function .

6. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 5, characterized in that: The hierarchical reinforcement learning model is adopted, and the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and the specific method for optimizing the hierarchical reinforcement learning model is: A hierarchical reinforcement learning model is used, wherein the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, and a channel cluster selection value function of the hierarchical reinforcement learning model is defined. , indicating the selection of channel clusters in state s The long-term value of the channel cluster selection strategy is designed based on the channel cluster selection value function. ; Define the channel selection Q value function , which means that in state s, from the channel cluster Select channel The long-term value of the channel selection strategy is designed based on the channel selection Q value function; Construct the loss function of the hierarchical reinforcement learning model, use the stochastic gradient descent algorithm to minimize the loss function, use the policy gradient method to maximize the channel cluster selection strategy objective function, and update the parameters of the channel cluster selection strategy , use the policy gradient method to maximize the channel selection strategy objective function and update the parameters of the channel selection strategy , and obtain the optimized hierarchical reinforcement learning model.

7. The method for dynamic channel configuration of multi-regional SMS service as claimed in claim 6, characterized in that: The specific method of obtaining the real-time SMS sending task, distributing the real-time SMS sending task channels based on the hierarchical reinforcement learning model, and sending the task is as follows: Obtaining a real-time SMS sending task, and extracting a business attribute feature vector of the real-time SMS sending task; The business attribute feature vector of the real-time SMS sending task is input into the hierarchical reinforcement learning model as the state s. The channel cluster selection strategy is used to make decisions on the first-layer action space, and the probability of selecting each channel cluster under the state s is calculated. The channel cluster selection strategy outputs the channel cluster selected in the first-layer action space. Input the state s and the selected channel cluster into the channel selection strategy, calculate the probability of selecting each channel under the state s and the selected channel cluster, make decisions on the second-layer action space through the channel selection strategy, and output the channel selected in the second-layer action space; Distribute real-time SMS sending tasks to the selected channels for sending.

8. A dynamic channel configuration system for multi-regional SMS services, which is implemented based on the dynamic channel configuration method for multi-regional SMS services according to any one of claims 1 to 7, characterized in that: include: The channel quality calculation module obtains the average delay, SMS sending success rate, and average cost of successfully sending 10,000 test SMS messages for each channel, calculates the price advantage coefficient between the unit price of the channel and the average unit price of other channels, and calculates the comprehensive quality index of each channel; The agent definition module defines the first-layer action space and the second-layer action space of the agent based on the comprehensive quality index, defines the state s as the business attribute feature vector of the SMS sending task, defines the action a of the agent as selecting a channel cluster from the first-layer action space and selecting a channel from the second-layer action space under the state s, and defines the first-layer reward function and the second-layer reward function; A reinforcement learning optimization module adopts a hierarchical reinforcement learning model, wherein the hierarchical reinforcement learning model includes a channel cluster selection strategy and a channel selection strategy, defines a channel cluster selection value function and designs a channel cluster selection strategy objective function, defines a channel selection Q value function and designs a channel selection strategy objective function, and optimizes the hierarchical reinforcement learning model; The real-time SMS sending module obtains the real-time SMS sending task, distributes the channel for the real-time SMS sending task based on the hierarchical reinforcement learning model, and sends it.

9. An electronic device, characterized in that: The invention comprises a memory, a processor and a computer program stored in the memory and executable on the processor. When the processor executes the program, the steps in the dynamic channel configuration method for multi-regional SMS service as described in any one of claims 1 to 7 are implemented.

10. A readable storage medium, characterized in that: The readable storage medium stores a computer program, and the computer program is suitable for being loaded by a processor to execute the steps in the dynamic channel configuration method for multi-region SMS service as described in any one of claims 1-7.

Images