Data transmission control method, device, computer device, and computer program

Abstract

The embodiments of the present application disclose a data transmission control method, an apparatus, a computer-readable storage medium, a computer device, and a computer program product, where the method is executed by a computer device, and includes the steps of: obtaining historical network performance data of each data transmission object among a plurality of data transmission objects, and calculating group network performance data of each object group among a plurality of object groups; performing weighting calculation on the group network performance data of each object group to obtain comprehensive network performance data of the plurality of data transmission objects; inputting the comprehensive network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; determining data transmission parameters of the object group according to the data transmission quality index and the group network performance data of each object group; when receiving a data transmission request sent from any one target data transmission object among the plurality of data transmission objects, determining target data transmission parameters according to the object group to which the target data transmission object belongs, and controlling a data transmission process corresponding to the data transmission request according to the target data transmission parameters.

Description

[Technical field]

[0001] The present application relates to the technical field of communications, including but not limited to a data transmission control method, an apparatus, a computer-readable storage medium, a computer device, and a computer program product.

[0002] This application is based on and claims priority to a Chinese patent application having application number 202210835843.8 and filing date July 15, 2022, the entire contents of which are hereby incorporated by reference into this application. [Background technology]

[0003] In recent years, with the thriving of various short video businesses, audio-video traffic has shown an explosive increase, and the contradiction between the ever-increasing needs of user experience at the client and the delay of network transmission performance has become a major contradiction that needs to be urgently resolved at present. In response to this, those skilled in the art have proposed a series of congestion control techniques, such as a cubic binary congestion control algorithm based on packet loss information (Cubic, Cubic Binary Increase Congestion) algorithm, a network congestion control algorithm based on bandwidth detection (BBR, Bottleneck Bandwidth and RTT), etc. The above congestion control algorithm can improve the quality of network transmission to a certain extent.

[0004] However, the congestion control algorithms in the related art are all control algorithms that conform to fixed rules and are set based on the experience of experts, and are difficult to adapt to the ever-changing network conditions and network business. Therefore, when facing some new scenarios, the current congestion control algorithms cannot adapt to these new scenarios, which leads to the problem of the network data transmission performance being reduced and the network performance being unstable. Summary of the Invention [Problem to be solved by the invention]

[0005] The embodiments of the present application provide a data transmission control method, an apparatus, a computer-readable storage medium, a computer device, and a computer program product, which can improve the data transmission performance of a network and the stability of the network performance.

[0006] An embodiment of the present application provides a data transmission control method, the method being executed by a computer device, and including the steps of: obtaining historical network performance data of each data transmission object among a plurality of data transmission objects, and calculating group network performance data of each object group among a plurality of object groups based on the historical network performance data, where the plurality of object groups are obtained by dividing the plurality of data transmission objects; performing weighting calculation on the group network performance data of each object group to obtain overall network performance data of the plurality of data transmission objects; inputting the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; determining data transmission parameters of a corresponding object group based on the data transmission quality index and the group network performance data of each object group; when receiving a data transmission request sent from any one target data transmission object among the plurality of data transmission objects, determining target data transmission parameters based on the object group to which the target data transmission object belongs, and controlling a data transmission process corresponding to the data transmission request based on the target data transmission parameters.

[0007] An embodiment of the present application provides a data transmission control device, comprising: an acquisition unit configured to acquire historical network performance data of each data transmission object of a plurality of data transmission objects, and calculate group network performance data of each object group of a plurality of object groups based on the historical network performance data, where the plurality of object groups are acquired by dividing the plurality of data transmission objects; a first calculation unit configured to perform weighting calculation on the group network performance data of each object group to acquire overall network performance data of the plurality of data transmission objects; and a second calculation unit configured to perform weighting calculation on the group network performance data of each object group to acquire overall network performance data of the plurality of data transmission objects. a learning unit configured to input group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; a second calculation unit configured to determine data transmission parameters of a corresponding object group based on the data transmission quality index and the group network performance data of each object group; and a control unit configured to determine target data transmission parameters based on an object group to which the target data transmission object belongs when receiving a data transmission request sent from a target data transmission object of any one of the plurality of data transmission objects, and to control a data transmission process corresponding to the data transmission request based on the target data transmission parameters.

[0008] An embodiment of the present application provides a computer-readable storage medium having a plurality of executable instructions stored thereon, the executable instructions being configured to implement the data transmission control method when executed by a processor.

[0009] An embodiment of the present application provides a computing device comprising a memory, a processor, and executable instructions stored in the memory and operable on the processor, the processor realizing the data transmission control method when executing the executable instructions.

[0010] An embodiment of the present application provides a computer program product, or a computer program, which includes executable instructions, the executable instructions being stored in a computer-readable storage medium, and which realizes the data transmission control method when a processor of a computing device reads the executable instructions from the computer-readable storage medium and executes the executable instructions.

[0011] The embodiment of the present application obtains historical network performance data of each data transmission object among a plurality of data transmission objects, and calculates group network performance data of each object group among a plurality of object groups according to the historical network performance data. The object group is a group obtained by dividing a plurality of data transmission objects. Then, a weighting calculation is performed on the group network performance data of each object group to obtain total network performance data, and a trained reinforcement learning model is called to predict based on the total network performance data and the group network performance data of each object group to obtain a data transmission quality index of each object group. Then, a data transmission parameter of each object group is determined according to the data transmission quality index and the group network performance data. Finally, when a data transmission request sent from any one of the target data transmission objects among the plurality of data transmission objects is received, a target data transmission parameter is determined according to the object group to which the target data transmission object belongs, and a data transmission process is controlled according to the target data transmission parameter. In this way, the data transmission control method provided in the present application performs group division on the data transmission objects, and then employs the recent historical network performance data of each object group for a certain period of time to perform reinforcement learning to obtain optimal network data transmission parameters for each object group at the current time. In this way, through the process of performing group division on multiple data transmission objects and performing reinforcement learning on the group network performance data of each object group, more accurate control of data transmission parameters can be realized.By matching optimal data transmission parameters for each data transmission object and making the currently determined data transmission parameters more suitable for the current network data transmission scene, the scene suitability of the data transmission control method and the data transmission performance of the network can be improved, and the stability of the network performance can be further improved. [Brief description of the drawings]

[0012] In order to more clearly describe the technical solutions in the embodiments of the present application, the following briefly introduces the drawings that need to be used in the description of the embodiments. It is obvious that the drawings in the following description are only some embodiments of the present application, and those skilled in the art can further obtain other drawings based on these drawings without creative labor.

[0013] [Figure 1] FIG. 2 is a schematic diagram of a scene of a data transmission control method provided by an embodiment of the present application; [Diagram 2] 2 is a flow diagram of a data transmission control method provided by an embodiment of the present application; [Diagram 3] FIG. 2 is a schematic diagram of the product side provided by an embodiment of the present application. [Figure 4] FIG. 2 is another flow diagram of a data transmission control method provided by an embodiment of the present application. [Figure 5a] FIG. 2 is a schematic diagram of dividing an IP address according to all rules provided by an embodiment of the present application; [Figure 5b] FIG. 2 is a schematic diagram of dividing an IP address according to a partial rule provided by an embodiment of the present application; [Figure 6] FIG. 2 is a flow diagram illustrating the measurement of QoS information by a network measurement module provided in an embodiment of the present application; [Figure 7] 2 is a schematic diagram of the processing flow of an intelligent learning module provided by an embodiment of the present application; [Figure 8] FIG. 2 is a schematic diagram of a processing process of a self-adaptive control module provided by an embodiment of the present application; [Figure 9]FIG. 2 is another flow diagram of a data transmission control method provided by an embodiment of the present application. [Figure 10] 1 is a structural schematic diagram of a data transmission control device provided by an embodiment of the present application; [Figure 11] FIG. 2 is a structural schematic diagram of a computer device provided by an embodiment of the present application; DETAILED DESCRIPTION OF THE PREFERRED EMBODIMENTS

[0014] The technical solutions in the embodiments of the present application are described below in a clear and complete manner in conjunction with the drawings in the embodiments of the present application. It is obvious that the described embodiments are only a part of the embodiments of the present application, but not all of the embodiments. Based on the embodiments of the present application, all other embodiments obtained by those skilled in the art without creative labor are all within the scope of protection of the present application.

[0015] The embodiments of the present application provide a data transmission control method, a device, a computer-readable storage medium, a computer device, and a computer program product. Here, the data transmission control method can be used in a data transmission control device. The data transmission control device can be integrated in a computer device, and the computer device can be a terminal or a server. Here, the terminal can be a device such as a mobile phone, a tablet PC, a notebook computer, a smart TV, a wearable smart device, a personal computer (PC, Personal Computer), and an in-vehicle terminal. The server can be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, and can be a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, safety services, network acceleration services (CDN, Content Delivery Network), and big data and artificial intelligence platforms. Here, the server can be a node in a blockchain.

[0016] Referring to FIG. 1, a schematic diagram of one scene of a data transmission control method provided by an embodiment of the present application. As shown in FIG. 1, the data transmission control system includes at least a gateway server A and a terminal B, and the terminal B can request the gateway server A to perform data transmission for specific data, that is, the terminal B can request the gateway server A to perform data transmission for the data to be transmitted by sending a data transmission request to the gateway server A. The gateway server A can obtain the requested data in response to the data transmission request of the terminal B, and return the requested data to the terminal B. Before receiving the data transmission request of the terminal B, the gateway server A can obtain historical network performance data of each data transmission object among the multiple data transmission objects, and calculate group network performance data of each object group according to the historical network performance data. The object group is a group obtained by dividing a plurality of data transmission objects, and performs a weighting calculation on the group network performance data of each object group to obtain comprehensive network performance data corresponding to the multiple data transmission objects. Then, based on the overall network performance data and the group network performance data of each object group, the trained reinforcement learning model is invoked to make predictions and obtain a data transmission quality index of each object group.Further, based on the data transmission quality index and the group network performance data of each object group, a data transmission parameter of each object group is calculated and obtained.In some embodiments, the data transmission control system may further include a data storage unit, and after calculating and obtaining the data transmission parameter of each object group, the data transmission parameter of each object group can be stored in the data storage unit.In this way, when gateway server A receives a data transmission request sent from terminal B, it can determine target data transmission parameters corresponding to the object group based on the object group to which terminal B belongs, and then control the data transmission process of terminal B based on the target data transmission parameters. That is, with terminal B as a data transmission object, it can determine which object group terminal B belongs to based on the terminal identifier of terminal B, then obtain data transmission parameters of the object group from the data storage unit, and determine the obtained data transmission parameters as the target data transmission parameters of terminal B. Then, it can control the data transmission process of terminal B based on the target data transmission parameters, and realize returning the requested data to terminal B.

[0017] For example, a short video application may be installed on terminal B. A user may perform a refresh operation through a client of the short video application, and the client of the short video application may generate a video recommendation request in response to the user's refresh operation, and send the video recommendation request to gateway server A as a data transmission request of the embodiment of the present application. Before responding to the video recommendation request sent by terminal B by the client of the short video application, gateway server A may adopt the data transmission control method provided by the embodiment of the present application to calculate and obtain the data transmission parameters of each object group. Here, the object groups may be different short video accounts in the short video application. When gateway server A responds to the video recommendation request sent by terminal B by the client of the short video application, gateway server A may obtain the short video accounts in the short video application of the user by analyzing the video recommendation request. There is a mapping relationship between the short video account and terminal B, so that the object group to which terminal B belongs can be determined according to the short video account, and the target data transmission parameters corresponding to the object group can be determined according to the object group to which terminal B belongs. Then, based on the target data transmission parameters, it can control a short video data transmission process of the video recommendation request sent by terminal B via the short video application client.

[0018] It is necessary to explain that the schematic diagram of the data transmission control scene shown in Figure 1 is only an example, and the data transmission control scene described in the embodiment of the present application is for more clearly explaining the technical solution of the embodiment of the present application, and does not constitute a limitation on the technical solution provided by the embodiment of the present application. Those skilled in the art can understand that as the data transmission control scene evolves and new business scenarios emerge, the technical solution provided by the embodiment of the present application can be similarly applied to similar technical problems.

[0019] Based on the above implementation scenarios, detailed explanations will be given below.

[0020] In the network data transmission scene, when the amount of audio-video data that needs to be transmitted is relatively large, the problem of congestion in data transmission will appear. In order to solve the congestion problem in data transmission and improve the data transmission efficiency and data transmission quality, those skilled in the art have proposed a number of congestion control algorithms to improve the network transmission quality. In this respect, as artificial intelligence technology has become increasingly mature in recent years, self-adaptive congestion control based on machine learning can automatically adapt to changes in network conditions and obtain benefits in network transmission performance through methods such as parameter setting. These methods can improve the network data transmission quality by setting relevant parameters of congestion control through the method of reinforcement learning. However, these methods are very difficult to implement because the parameter set is set by artificial experience and these methods are completely deviated from the existing congestion control mechanism. In addition, these methods have the problem of lack of stability, making them unable to be deployed and used in current networks. In contrast, the embodiments of the present application are expected to provide a data transmission control method to alleviate the congestion problem in data transmission to a certain extent and improve data transmission performance.

[0021] The embodiment of the present application is described from the viewpoint of a data transmission control device, which can be integrated in a computer device. Here, the computer device can be a terminal or a server. As shown in FIG. 2, which is a flow diagram of a data transmission control method provided by an embodiment of the present application, the embodiment of the present application is described by taking as an example that the data transmission control method is executed by a server in a computer device, and the method includes the following steps S101 to S105.

[0022] Step S101: Obtain historical network performance data of each data transmission object of the plurality of data transmission objects, and calculate group network performance data of each object group of the plurality of object groups based on the historical network performance data.

[0023] In the embodiment of the present application, the plurality of data transmission objects may be all registered users of a certain application program. For example, an instant messaging application program or a short video application program both have an application server providing data services and a plurality of application clients that register the application. Each application client may be an object that the application server of the application program performs data transmission, so it can be determined that all application clients that register the application program are a plurality of data transmission objects. Or, in some circumstances, some clients that register the application program may be set to be a plurality of data transmission objects. For example, in the above-mentioned short video application, if only some registered users' clients among the application clients of the short video application are member user clients, data transmission control can be performed only for these member user clients, so that the data transmission performance when these member user clients perform data transmission can be optimized, and good data transmission service and smooth viewing experience can be provided for these member user clients. In addition, or in some circumstances, if some users have a relatively low need for data transmission, while other users have a relatively high need for data transmission, then these users with a relatively high need for data transmission can be optimized appropriately and determined as a plurality of data transmission objects, where the data transmission need parameters of the users can be determined according to the frequency of the users' data transmission and the amount of data requested to be transmitted, and when the data transmission need parameter of any one of the users is greater than a need parameter threshold, it represents that the data transmission need of the user is relatively high, and when the data transmission need parameter of any one of the users is less than or equal to the need parameter threshold, it represents that the data transmission need of the user is relatively low.That is, in the embodiment of the present application, personalization settings can be made for multiple data transmission objects, but this is not limited thereto.

[0024] The historical network performance data of a data transmission object may be data obtained after collecting network performance data for a certain period of time prior to the current time of the data transmission object. For example, statistics can be collected periodically for the network performance data of the data transmission object. For example, statistics are collected every T time interval for network performance data for the time period between the previous statistics time and the current statistics time. When it is necessary to control data transmission, the network performance data of the data transmission object for which statistics have been collected most recently can be used as the historical network performance data of the data transmission object.

[0025] In some embodiments, the network performance data of a data transmission object is a parameter used to reflect the network transmission performance when the data transmission object performs data transmission, and the network performance data of the data transmission object can characterize the network quality of service information (QoS, Quality of Service) when the data transmission object performs data transmission. The historical network performance data of a data transmission object refers to the network performance data within a historical period, and is a parameter used to reflect the network transmission performance when the data transmission object performs data transmission within the historical period.

[0026] For example, the network performance data may be QoS information such as the minimum round trip time (RTT) (which may be represented as min_rtt), the maximum round trip time (which may be represented as max_rtt), the smoothed round trip time (which may be represented as srtt), the maximum movement amount (which may be represented as max_inflight), the effective throughput (which may be represented as goodput), the retransmission rate (which may be represented as retran_ratio), the initial window (which may be represented as init_cwnd), and the maximum window (which may be represented as max_cwnd) when the data transmission object transmits data.

[0027] In some embodiments, obtaining the historical network performance data of a plurality of data transmission objects and calculating the group network performance data of each object group based on the historical network performance data can be realized in the following manner: First, obtain the Internet Protocol address data of each data transmission object among the plurality of data transmission objects, and obtain the historical network performance data of each data transmission object; Second, divide the plurality of data transmission objects into a plurality of object groups based on the Internet Protocol address data of each data transmission object; Finally, calculate the group network performance data of the object group based on the historical network performance data of the data transmission objects in each object group.

[0028] In the embodiment of the present application, the object group refers to a group obtained after grouping a plurality of data transmission objects, and a plurality of data transmission objects can be divided into a plurality of groups according to any one grouping manner. Each group can be called an object group. In this way, the historical network performance data corresponding to each object group can be calculated based on the historical network performance data of the data transmission objects in each object group, that is, the group network performance data of each object group can be calculated and obtained.

[0029] In the embodiment of the present application, the group division of the multiple data transmission objects may be based on the Internet Protocol address data of each data transmission object. For example, the Internet Protocol address data may be the IP address of the data transmission object. As described above, if the data transmission object is a client of an application program, the IP address of the data transmission object may be the IP address of a terminal on which the client is installed. The terminal may be any terminal capable of connecting to the Internet and transmitting data, such as a smartphone, a tablet, a smart watch, or a personal computer.

[0030] For example, first, the IP addresses of a plurality of data transmission objects and the historical network performance data of each data transmission object can be obtained. Next, the plurality of data transmission objects can be divided into a plurality of object groups based on the IP addresses of each data transmission object, for example, the data transmission objects corresponding to the IP addresses belonging to a preset address range can be divided into one object group. That is, the IP addresses of the data transmission objects can be divided based on the preset address range, and the data transmission objects corresponding to the IP addresses belonging to the same preset address range can be divided into one object group. Next, the group network performance data of each object group is calculated based on the historical network performance data of the data transmission objects included in each object group. Here, the group division based on the IP addresses of the data transmission objects may be division based on rules such as the region of the IP address, the operator number, the autonomous system (AS) number, the access method, etc. When the group division is performed according to the IP address data of the data transmission objects, the division may be performed according to the entire rule, or may be performed according to a partial rule. For example, when dividing according to the whole rule, it can be divided according to a method such as region + operator + AS number + access method, and when dividing according to a partial rule, it can be divided according to rules such as region + operator, region + access method, or region. It can be understood that the data transmission process of the data transmission object in the same object group has the same network forwarding path and also has the same network state information. That is, the data transmission object of the same object group can adopt the same data transmission parameter to control when performing data transmission control.

[0031] In some embodiments, calculating the group network performance data of each object group based on the historical network performance data of the data transmission objects in each object group can be realized in the following manner: first, determine a historical network performance data set corresponding to the corresponding object group based on the historical network performance data of each data transmission object in each object group, then calculate an average value of the historical network performance data in each historical network performance data set, and determine the average value corresponding to each historical network performance data set as the group network performance data of the corresponding object group.

[0032] In the embodiment of the present application, the calculation of the group network performance data corresponding to each object group can average the historical network performance data of the data transmission objects included in each object group to obtain the average value of the historical network performance data, and determine the average value of the historical network performance data as the group network performance data corresponding to the corresponding object group. For example, the data transmission objects included in each object group can be determined first, and then the historical network performance data of each data transmission object in the object group can be determined to obtain a historical network performance data set corresponding to each object group. Then, the average value of the historical network performance data in each historical network performance data set can be calculated. That is, all the historical network performance data in the historical network performance data set is added, and then the result of the addition is divided by the number of historical network performance data in the historical network performance data set to obtain the average value of the historical network performance data in the historical network performance data set, thereby obtaining the group network performance data corresponding to each object group.

[0033] In some embodiments, before calculating the average value of the historical network performance data of a plurality of data transmission objects in each object group and calculating the group network performance data of the object group, anomaly detection can be performed on the historical network performance data of each data transmission object first to determine whether abnormal data exists in the historical network performance data of the data transmission object in the object group. This can avoid the existence of abnormal data in the historical network performance data of the data transmission object in the object group, thereby ensuring the accuracy of the calculation of the group network performance data. Here, when performing anomaly detection on the historical network performance data of each data transmission object, the detection can be based on historical network performance data on a time dimension with a preset length, and the preset length is greater than the time length corresponding to the historical network performance data. For example, if the historical network performance data of the data transmission object is QoS data that dates back within one hour from the current time, QoS data that dates back within one month from the current time of the data transmission object can be adopted to detect the QoS data that dates back within one hour for the data transmission object, thereby determining whether an obvious abnormality occurs in the QoS data. In the process of implementation, based on the QoS data of the data transmission object that is within one month from the current time, the QoS data of the data transmission object that is within one hour from the current time is predicted to obtain the predicted QoS data. Then, the difference between the predicted QoS data and the actual QoS statistics is calculated, and if the difference rate is greater than a preset ratio, it can be determined that the historical network performance data of the data transmission object is abnormal.

[0034] If it is determined that an anomaly exists in the QoS data of a data transmission object, the historical network performance data of the data transmission object may be removed when performing the calculation of the group network performance data.

[0035] Step S102: Perform weighting calculation on the group network performance data of each object group to obtain integrated network performance data of the multiple data transmission objects.

[0036] Here, after calculating and obtaining group network performance data for each object group, a weighting calculation is further performed on the group network performance data of all object groups to obtain overall network performance data for all data transmission objects, which can be referred to as overall network performance data here.

[0037] In some embodiments, performing weighting calculation on the group network performance data to obtain the overall network performance data can be realized in the following manner: first, obtain the weighting coefficient of each object group; then perform weighting calculation on the group network performance data of each object group according to the weighting coefficient to obtain the overall network performance data corresponding to a plurality of data transmission objects.

[0038] In an embodiment of the present application, to calculate the overall network performance data based on the group network performance data of each object group, a weighting coefficient corresponding to each object group is first obtained, and then a weighting calculation is performed on the group network performance data of the object group based on the weighting coefficient of each object group, thereby obtaining the overall network performance data.

[0039] In some embodiments, the weight coefficient of each object group can be obtained by the following manner: first, obtain the quantity of data transmission objects contained in each object group; then calculate the ratio between the quantity of data transmission objects and the quantity of the plurality of data transmission objects in each group to obtain the weight coefficient of each object group; that is, determine the ratio between the quantity of data transmission objects and the quantity of the plurality of data transmission objects in each object group as the weight coefficient of the corresponding object group.

[0040] In the embodiment of the present application, the weight coefficient of each object group can be calculated and obtained based on the number of data transmission objects contained in the corresponding object group. For example, the number of data transmission objects contained in each object group can be obtained first, and then the ratio between the number of data transmission objects in each group and the number of the plurality of data transmission objects can be calculated. That is, the ratio of the number of members in each group to the total number of members can be calculated, and then the weight coefficient of each object group can be obtained.

[0041] Step S103: The overall network performance data and the group network performance data of each object group are input into the trained reinforcement learning model to obtain a data transmission quality index of each object group.

[0042] Here, the reinforcement learning model (RL) is one of the models in the field of machine learning, and the reinforcement learning model is a data processing model realized based on reinforcement learning. Reinforcement learning is inspired by behavioral psychology, and mainly focuses on how an agent takes different actions in the environment to maximize cumulative rewards. Reinforcement learning mainly consists of an agent, an environment, a state, an action, and a reward. After an agent performs a certain action, the environment is transformed into a new state, and a reward signal (positive reward or negative reward) is given to the new state environment. After that, the agent performs a new action according to a certain policy based on the new state and the reward fed back by the environment. The above process is a method in which the agent and the environment interact with each other through the state, action, and reward. Through reinforcement learning, the agent can know in what state and what action it should take to obtain the maximum reward. Because the interaction method between the agent and the environment is similar to that between humans and the environment, reinforcement learning can be regarded as a set of general learning frameworks and can be used to solve problems in general artificial intelligence. Therefore, reinforcement learning is also called a machine learning method for general artificial intelligence.

[0043] In the embodiment of the present application, the setting of network performance data in the data transmission process can be controlled according to the above characteristics of the reinforcement learning model. For example, the corresponding reward and environmental state can be set according to the overall network performance data and the group network performance data of each object group, so that the pre-defined reinforcement learning model can learn a data transmission quality index corresponding to each object group. The data transmission quality index can be used to obtain a better reward value by controlling the data transmission parameters of the corresponding object group. That is, the data transmission quality index is data output by the reinforcement learning model and used to control the data transmission parameters of the corresponding object group.

[0044] In some embodiments, invoking the trained reinforcement learning model to predict based on the overall network performance data and the group network performance data of each object group to obtain the data transmission quality index of each object group may be performing reinforcement learning on the overall network performance data and the group network performance data of each object group based on the reinforcement learning model, thereby obtaining the data transmission quality index of each object group. In some embodiments, the data transmission quality index of each object group can be obtained by the following manner: First, calculate the reinforcement learning parameters of each object group based on the group network performance data of each object group. Then, use the overall network performance data as the state value of the reinforcement learning model, and input the reinforcement learning parameters as the reward value of the reinforcement learning model into the reinforcement learning model, and obtain the output data transmission quality index of each object group.

[0045] In some embodiments, the trained reinforcement learning model may be any type of reinforcement learning model, such as a deep reinforcement learning (DRL) model, a search and supervision based DRL model, a value function based DRL model, a policy gradient based DRL model, etc.

[0046] In the embodiment of the present application, first, the reinforcement learning parameters of each object group are calculated according to the group network performance data of each object group, and the reinforcement learning parameters are used as the reward values ​​of the reinforcement learning model. Then, the above-mentioned calculated and acquired overall network performance data are used as the state values ​​of the reinforcement learning model, and the reward values ​​and the state values ​​are input into the reinforcement learning model, and the reinforcement learning process is performed on the reinforcement learning model according to the above-mentioned reward values ​​and state values, and the output parameters learned and acquired by the reinforcement learning model are obtained. That is, the data transmission quality index of each object group can be output by the reinforcement learning model.

[0047] As can be understood, for different object groups, the reinforcement learning parameters calculated and obtained by the reinforcement learning model are different, i.e., when the reward value of the reinforcement learning model is different, the learned output parameters are not the same. That is, for different object groups, the reinforcement learning model can learn different output parameters. The output parameters can control the data transmission parameters of the corresponding object groups, so that the reinforcement learning parameters of the object groups continue to improve, and therefore the output parameters can also be called data transmission quality indexes.

[0048] In some embodiments, calculating the reinforcement learning parameters of each object group based on the group network performance data of each object group can be realized by the following manner: First, extract effective throughput data and smoothed round-trip transmission time data in the group network performance data of each object group; Then, calculate the ratio between the effective throughput data corresponding to each object group and the corresponding smoothed round-trip transmission time data, and determine the ratio as the reinforcement learning parameters of each object group.

[0049] In the embodiment of the present application, the reinforcement learning parameters of each object group can be calculated by adopting the effective throughput data and the smoothed round trip transmission time data in the group network performance data of each object group. First, the effective throughput data and the smoothed round trip transmission time data are extracted from the group network performance data of each object group. Then, the ratio between the effective throughput data and the smoothed round trip transmission time data can be calculated, and the ratio can be used as the reinforcement learning parameter of the corresponding object group. In the embodiment of the present application, setting the ratio between the effective throughput data and the smoothed round trip transmission time data as the reinforcement learning parameter of the object group can also be using the ratio as the reward value of the reinforcement learning model. Here, it is mainly considered that the learning target of the reinforcement learning model obtains a higher reward value. That is, it is desirable to obtain a larger effective throughput and a smaller smoothed round trip transmission time, that is, in order to obtain a better QoS, the ratio between the effective throughput data and the smoothed round trip transmission time data is used as the reward value of the reinforcement learning model.

[0050] Here, it is necessary to explain that throughput refers to the number of bits received per second, not including control data (when there are no frames, the device will reduce the maximum rate). Effective throughput data (goodput) refers to the number of bits sent to the correct target interface per unit time, which is discarded or retransmitted when reduced. In computer networks (including wireless networks), goodput is the throughput at the application program level, i.e., each unit is expressed using bits, and the number of transfers from a source address to a destination on a network does not include protocol overhead and retransmission data of data packets. For example, when one file is transferred, the effective throughput of the user experience corresponds to dividing the time of file transmission into the size of the bit file. When the effective throughput is generally lower than the throughput (total bit rate transmission physical), it is usually lower than the network access connection speed (channel capacity and bandwidth). Effective throughput data refers to the amount of data transmitted on the network, and the effective throughput data can evaluate the performance of the network, and the effective throughput data refers to the amount of effective data transmitted on the network.

[0051] The smoothed round trip transmission time data refers to the weighted average round trip time, where the round trip time refers to the difference between the time when the message segment is issued and the time when the corresponding confirmation is received. The calculation method of the round trip time is that there is a time stamp option in the option of the TCP message header, where there is a time stamp value and a time stamp loopback field, and the round trip time can be calculated based on the time stamp value and the time stamp loopback field. The weighted average round trip time, also called the smoothed round trip time, is the round trip time obtained after applying an average weight to the round trip time.

[0052] In some embodiments, adopting a reinforcement learning model to learn the data transmission quality index of each object group may be adopting a reinforcement learning model to perform reinforcement learning multiple times based on the overall network performance data and the group network performance data of each object group to obtain multiple data transmission quality indexes of each object group. Then, further calculate the average value of the multiple data transmission quality indexes, and finally obtain a more accurate data transmission quality index of each object group. In the embodiments of the present application, the average value of the output parameters obtained by multiple learning is taken as the final output parameter, thereby avoiding the problem that the learned data transmission quality index is inaccurate due to learning error, and thereby improving the accuracy of the learned data transmission quality index.

[0053] Step S104: Determine data transmission parameters of corresponding object groups according to the data transmission quality index and the group network performance data of each object group.

[0054] After adopting a reinforcement learning model to learn and obtain the data transmission quality index of each object group, the group network performance data of each object group can be adjusted according to the data transmission quality index of each object group to obtain the data transmission parameters of each object group. By controlling the data transmission process of the corresponding object group with the data transmission parameters of each object group, a better data transmission effect can be obtained, that is, the data transmission performance can be improved.

[0055] In the embodiment of the present application, the data transmission parameters are control parameters used to control the data transmission process. In the data transmission process, the network bandwidth of the corresponding data transmission object is controlled according to the data transmission parameters of the data transmission object, so as to avoid network congestion and ensure the transmission performance of the entire network.

[0056] In some embodiments, calculating and obtaining the data transmission parameters of each object group based on the data transmission quality index and the group network performance data can be realized in the following manner: First, extract the effective throughput data and the minimum round trip transmission time data of each object group from the group network performance data; Then, calculate the initial window data of the corresponding object group based on the effective throughput data, the minimum round trip transmission time data and the data transmission quality index of each object group, and determine the data transmission parameters of the corresponding object group based on the initial window data of each object group.

[0057] In the embodiment of the present application, adjusting the group network performance data of the corresponding object group according to the learned data transmission quality index of each object group may be first extracting effective throughput data and minimum round trip transmission time data from the group network performance data of each object group, then calculating and obtaining initial window data of each object group according to the effective throughput data, the minimum round trip transmission time data and the learned data transmission quality index, and further determining the initial window data of each object group as the data transmission parameter of the corresponding object group. In some embodiments, the data transmission parameters of each object group can further be associated with the corresponding object group and stored. In this way, when it is necessary to use the data transmission parameters to perform data transmission control, the data transmission parameters of the object group can be queried from the stored data, thereby improving the efficiency of data transmission.

[0058] Step S105: When receiving a data transmission request sent from a target data transmission object among the multiple data transmission objects, determine target data transmission parameters based on an object group to which the target data transmission object belongs, and control a data transmission process corresponding to the data transmission request based on the target data transmission parameters.

[0059] After determining the data transmission parameters corresponding to each object group, and before completing the statistics of the historical network performance data for the next cycle, the data transmission process of the corresponding data transmission object can be controlled based on the data transmission parameters corresponding to each object group.

[0060] In the process of realization, when a data transmission request sent from a target data transmission object is received, first determine an object group corresponding to the target data transmission object, then obtain target data transmission parameters of the object group, and control the data transmission process of the target data transmission object according to the target data transmission parameters.For example, when a data connection request sent by a client of an application to an application server is received, determine the target object group to which the client belongs, and determine initial window data of the target object group, and then use the initial window data to control the data transmission process for the client.

[0061] As can be seen from the above description, the data transmission control method provided by the embodiment of the present application obtains historical network performance data of each data transmission object among a plurality of data transmission objects, and calculates group network performance data of each object group based on the historical network performance data. The object group is a group obtained by dividing a plurality of data transmission objects. Then, a weighting calculation is performed on the group network performance data to obtain overall network performance data, and based on the overall network performance data and the group network performance data of each object group, a reinforcement learning model after training is called for prediction to obtain a data transmission quality index of each object group. Then, a data transmission parameter of each object group is calculated and obtained based on the data transmission quality index and the group network performance data. In this way, when a data transmission request sent from a target data transmission object is received, a target data transmission parameter is determined based on the object group to which the target data transmission object belongs, and a data transmission process is controlled based on the target data transmission parameter. In this way, the data transmission control method provided by the embodiment of the present application performs group division on the data transmission objects, and then employs the recent historical network performance data of each object group for a certain period of time to perform reinforcement learning to obtain an optimal network data transmission parameter at the current time of each object group. The method provides fine-grained control over network data transmission parameters, so as to match optimal network data transmission parameters at the current time for each data transmission object. In this way, the data transmission performance of the data transmission object can be greatly improved. In addition, the data transmission control method provided by the embodiment of the present application can provide finer and more accurate data transmission control, and the session parameters can be more accurately set when a connection is established.Such a method of setting data transmission control parameter values ​​that match the network characteristics for each object group is advantageous in improving the transmission quality of the entire network system. That is, by performing group division for a plurality of data transmission objects and performing reinforcement learning on the group network performance data of each object group, more accurate control of data transmission parameters can be realized. By matching optimal data transmission parameters for each data transmission object and making the currently determined data transmission parameters more suitable for the current network data transmission scene, the scene suitability of the data transmission control method and the data transmission performance of the network can be improved, and the stability of the network performance can be further improved.

[0062] It is necessary to explain that in the embodiment of the present application, the content of the user information includes, for example, the historical network performance data of each user (i.e., the historical network performance data of each data transmission object), the user group (i.e., the object group) to which each user belongs, the data transmission parameters of each object group, etc. If it is related to user information or enterprise information, the permission or consent of the user needs to be obtained when the embodiment of the present application is applied in a specific product or technology. In addition, the collection and processing of related data should be strictly in accordance with the requirements of the laws and regulations of the relevant countries, and the informed consent or independent consent of the personal information subject should be obtained when applying the embodiment, and the subsequent data use and processing should be carried out within the scope of the laws and regulations and the authorization of the personal information subject.

[0063] In some embodiments, the data transmission control method may be a self-adaptive congestion control technique divided for user groups in a network communication system, which belongs to the field of optimizing network traffic transmission performance. In the embodiments of the present application, the server first divides the IP addresses of clients into different object groups according to information such as regions and operators, and takes statistics on the network performance data of the previous time cycle of the network stream of each object group at regular intervals. Based on the network performance data of the previous cycle collected by the server, the server outputs the network transmission parameter values ​​(e.g., initial transmission window, initial transmission rate, etc.) of each object group to be applied to this cycle by expert experience or artificial intelligence method. When a new connection request from a client arrives at the server, the server queries and obtains the object group identifier and network transmission parameters corresponding to the client according to the IP address of the client. The server responds to the client's request and sends the corresponding network data according to the network transmission parameters. The self-adaptive congestion control technique of the embodiments of the present application has advantages such as high stability, precise control, high self-adaptation, and light weight, which is advantageous in ensuring feasibility and at the same time improving the transmission performance of the entire network system.

[0064] The purpose of the embodiment of the present application is to solve the difficult problem that the current network traffic cannot realize self-adaptive congestion control, while simultaneously achieving advantages such as stability, precise control, etc. In the case of the self-adaptive congestion control technology for user groups (i.e., the above-mentioned object groups) proposed in the embodiment of the present application, the technology effectively solves the stability problem encountered in the execution process of machine learning by combining the historical information measured by the network with reinforcement learning, and sets specific initialization network parameters for each user group by the manner of dividing the user group and measuring the network for the user group, thereby realizing precise control for the initial stage of network traffic transmission.

[0065] Figure 3 is a schematic diagram of a product side provided by an embodiment of the present application. As shown in Figure 3, a cloud server 301A is connected to a gateway server 302A, which divides client IPs into different user groups (UG, User Group) 303A according to different rules such as regions, operators, access methods, etc., and sets dedicated congestion control parameters for the sessions of the user groups by a reinforcement learning method according to the latest network state. When the gateway server 302A receives a connection request from a client, it can obtain (initialized) congestion control parameters suitable for the user group by inquiring about the user group identifier corresponding to the client IP, and respond to the client request based on the congestion control parameters.

[0066] The embodiment of the present application further provides a data transmission control method, which can be used in a computer device, and the computer device can be a server. As shown in Figure 4, which is another flow diagram of the data transmission control method provided by the embodiment of the present application, the method includes the following steps S201 to S209:

[0067] Step S201: The server divides users into groups and obtains a plurality of user groups.

[0068] Here, the server can divide the IP addresses of clients corresponding to users according to rules such as region, operator, AS number, access method, etc., and divide the IP addresses of clients into different user groups. Here, the connections (or sessions) of each user group have similar network forwarding paths and similar network status information. The division of user groups is not limited to division for all client IP addresses, and may be division for some specific IP address prefixes, or some IP address prefixes whose traffic ratios are greater than a traffic ratio threshold.

[0069] In some embodiments, the rule for dividing user groups may be divided according to the whole rule (i.e., region, operator, AS number, access method, etc.), or may be divided according to a partial rule, such as region + operator, region + access method, region. Figure 5a and Figure 5b are schematic diagrams of solutions for dividing IP addresses of user clients. Here, Figure 5a divides according to the whole rule, and Figure 5b divides according to a partial rule (region).

[0070] [Table 1]

[0071] Table 1 is a user group division information table, and user groups can be divided according to IP address prefixes. All IP addresses belonging to the same user group have the same user group identifier.

[0072] Step S202: The server takes statistics on the QoS data within the statistical cycle, and calculates and obtains the QoS data of each user group according to the taken QoS data.

[0073] In an embodiment of the present application, the server can record the QoS information (i.e., QoS data) of the network for each data stream and the IP address of the client. For example, when each stream is finished, the server will write the relevant QoS information of this stream and the IP address of the client in the server's log file. The QoS information includes information such as minimum RTT (min_rtt), maximum RTT (max_rtt), smoothed RTT (srtt), maximum inflight (max_inflight), effective throughput (goodput), retransmission rate (retran_ratio), initial window (init_cwnd), maximum window (max_cwnd), etc. That is, the QoS information indicates the service quality of the network. After completing the transmission of each data stream, the server records information such as the IP address of the data stream, as well as the minimum RTT (min_rtt), maximum RTT (max_rtt), smoothed RTT (srtt), maximum movement amount (max_inflight), effective throughput (goodput), retransmission rate (retran_ratio), initial window (init_cwnd), maximum window (max_cwnd), etc. of the data stream, thereby realizing the acquisition of QoS information of the data stream.

[0074] The server can collect statistics on the QoS average value and related quantile value (e.g., 50th or 90th quantile QoS value) of each user group in a certain period (statistical cycle T). For example, the server side reads the QoS information of each stream and the IP address of the client one by one from the log file, and obtains the user group identifier of each data stream by inquiring about the correspondence between the IP address and the user group identifier. The server side then obtains the average value or related quantile value (e.g., 50th or 90th quantile) of the QoS information of data streams having the same user group identifier.

[0075] For example, if the time cycle T is 15 mins, there are 100 sessions with IP address 192.168.0.0 / 24, 200 sessions each with IP addresses 192.168.1.0 / 24 and 192.168.2.0 / 24, and 50 sessions each with IP addresses 192.168.3.0 / 24 and 192.168.4.0 / 24 during this period. Therefore, the number of sessions for user group 1 (UG_1) is 500, and the number of sessions for user group 2 (UG_2) is 100. In the embodiment of the present application, the QoS average values ​​(QoS_avg_1) of the 500 sessions in UG_1 are the minimum RTT average value (min_rtt_avg_1), the maximum RTT average value (max_rtt_avg_1), the smoothed RTT average value (srtt_avg_1), the effective throughput average value (goodput_avg_1), the retransmission rate average value (retran_ratio_avg_1), the initial window average value (init_cwnd_avg_1), and the maximum window average value (max_cwnd_avg_1), respectively. Assume that the QoS averages (QoS_avg_2) of the 100 sessions in UG_2 are the minimum RTT average (min_rtt_avg_2), the maximum RTT average (max_rtt_avg_2), the smoothed RTT average (srtt_avg_2), the effective throughput average (goodput_avg_2), the retransmission rate average (retran_ratio_avg_2), the initial window average (init_cwnd_avg_2), and the maximum window average (max_cwnd_avg_2), respectively. Therefore, the reward value (reward) of UG_1 is goodput_avg_1 / srtt_avg_1, and the reward value of UG_2 is goodput_avg_2 / srtt_avg_2.

[0076] In some embodiments, measuring and taking statistics of the QoS information of the previous time cycle of each user group at regular intervals can be realized by a network measurement module in the server. Figure 6 is a flow diagram of the network measurement module provided by the embodiment of the present application for measuring QoS information. As shown in Figure 6, firstly, obtain session log information 601 and user group information 602, and then perform network measurement and data statistics 603 on the obtained session log information 601 and user group information 602, thereby obtaining the QoS information of different user groups. For example, the QoS information of user group 1, QoS information of user group 2, . . . QoS information of user group n shown in Figure 6, where n is an integer greater than 2.

[0077] Step S203: The server performs weighting calculation on the QoS data of each user group, and sets the data obtained by the weighting calculation as the state value of the reinforcement learning model.

[0078] Furthermore, the server can obtain a weighting factor for each user group, which may be manually set by a user or calculated based on the number of users in the user group. After determining the weighting factor for each user group, the server can perform weighting calculation on the QoS data of each user group based on the weighting factor of each user group, and the result obtained by the weighting calculation can be the state value of the reinforcement learning model.

[0079] In some embodiments, the intelligent learning module in the server can perform weighting calculation on the QoS data of each user group. In some other embodiments, the intelligent learning module can be located in the server, or the intelligent learning module can be located in the gateway or in a dedicated server at the far end. The intelligent learning module includes a reinforcement learning model, which can realize the control of the initialization parameters transmitted for the traffic of multiple user groups. That is, the reinforcement learning model can set congestion control parameters for the traffic of multiple user groups.

[0080] In the embodiment of the present application, the server weights the QoS information according to the user group, where the QoS data of the user group i is denoted as QoS_i, and the weight of QoS_i is denoted as w_i. At this time, the QoS after weighting is shown in formula (1).

[0081]

number

[0082] Step S204: The server extracts the effective throughput data and the smoothed round-trip transmission time data in each user group QoS data, calculates a ratio between the effective throughput data and the smoothed round-trip transmission time data, and determines the ratio as a reward value of the reinforcement learning model.

[0083] In the embodiment of the present application, the server extracts the effective throughput data goodput and the smoothed round trip transmission time data srtt from the QoS data of each user group, calculates the ratio between the effective throughput data goodput and the smoothed round trip transmission time data srtt, and determines the ratio as the reward value of the reinforcement learning model. Here, the calculation formula of the reward value Reward of the reinforcement learning model is shown in the following formula (2).

[0084]

number

[0085] In some embodiments, the effective throughput data (goodput) in formula (2) can be replaced with throughput (throughput), or can be further replaced with the download rate (download_speed) of data from the application layer to the network layer. In some other embodiments, the effective throughput data (goodput) in formula (2) can be further replaced with the average RTT, the minimum RTT, or the maximum RTT.

[0086] Step S205: The server performs reinforcement learning on the state values ​​and the reward values ​​corresponding to each user group according to the reinforcement learning model, and obtains learning parameters corresponding to each user group output by the reinforcement learning model.

[0087] After the state value and the reward value of the reinforcement learning model are determined, the state value and the reward value can be input to the reinforcement learning model to perform reinforcement learning, and a parameter α output by the reinforcement learning model can be obtained. Here, the reinforcement learning model here can be one reinforcement learning model or multiple reinforcement learning models. Since different user groups provide different reward values ​​to the reinforcement learning model, the learning parameters learned by the reinforcement learning model for different user groups are not the same. For example, the output of the reinforcement learning model can be one of a parameter set {-1, -0.5, 0, 0.5, 1} (denoted as α, with a default value of 0).

[0088] 7 is a flow diagram of the process of the intelligent learning module provided by the embodiment of the present application. As shown in FIG. 7, the input parameters of the reinforcement learning model 701 input in the intelligent learning module 700 are user group weighted QoS data 702. The user group weighted QoS data 702 includes, but is not limited to, RTT, effective throughput data (goodput), maximum movement amount (max_inflight), and reward value. The output of the reinforcement learning model 701 is a parameter α.

[0089] Step S206: The server calculates data transmission parameters corresponding to the corresponding user group according to the learning parameters corresponding to each user group and the QoS information of each user group.

[0090] Here, after learning and acquiring the learning parameters corresponding to each user group, data transmission parameters corresponding to each user group can be further calculated based on the learning parameters and QoS information of each user group. Here, the data transmission parameters can also be called initialized congestion control parameters, and the data transmission parameters of each user group include an initial transmission window init_cwnd and an initial RTT value init_rtt, where the initial transmission window init_cwnd and the initial RTT value init_rtt can be calculated according to the following formula (3) and formula (4).

[0091]

number

[0092] For example, for the above user group 1, i.e., UG_1, the calculation formulas for the initialized congestion control parameters of user group 1, i.e., the initial transmission window init_cwnd_1 of user group 1 and the initial RTT value init_rtt_1 of user group 1, are shown in the following equations (5) and (6).

[0093]

number

[0094] Here, min_rtt_avg_1 represents the minimum RTT average value of user group 1, and goodput_avg_1 represents the effective throughput average value of user group 1.

[0095] For each user group, the above method can be used to calculate and obtain the data transmission parameters of the user group. Then, the data transmission parameters of each user group can be stored for use. Table 2 shows a schematic diagram of the stored initialization parameter list.

[0096] [Table 2]

[0097] In the embodiment of the present application, the calculation process of the data transmission parameters can be realized by a self-adaptation control module. Figure 8 is a schematic diagram of the processing process of the self-adaptation control module provided in the embodiment of the present application. As shown in Figure 8, the self-adaptation control module calculates the initial congestion parameter value applied to each user group (UG_i) for the user group according to the QoS information (QoS_i) of the user group i and the parameter α through a parameter calculation method. In the embodiment of the present application, the initial congestion parameters of all user groups jointly constitute an initial parameter list, for example, init_rtt and init_cwnd corresponding to UG_1, UG_2 ... UG_n in the finally obtained data transmission parameters in Figure 8.

[0098] Step S207: When the server receives a data transmission request sent from the user client, it obtains the IP address of the user client.

[0099] When the server receives a data transmission request sent from a user client, it can obtain the IP address included in the connection request.

[0100] Step S208: The server determines corresponding group information according to the IP address of the user client, and determines data transmission parameters corresponding to the group information.

[0101] After obtaining the IP address of the user client, further determine group information corresponding to the user client, and further find the initialization parameters corresponding to the group in the initialization parameter list according to the group information, i.e., find the corresponding data transmission parameters.

[0102] Step S209: The server responds to the connection request of the user client based on the corresponding data transmission parameters, and sends data to the user client.

[0103] In the embodiment of the present application, when the server side receives a data transmission request (for example, it may be a session request) from a user client, it first inquires a user group identifier (UG_i) based on the IP address of the user client. Then, it inquires an initial parameter list based on the obtained UG_i, and obtains the initialization congestion parameters init_rtt and init_cwnd corresponding to UG_i. Finally, the server responds to the data transmission request of the user client with init_rtt and init_cwnd, and sends the data.

[0104] After determining the data transmission parameters of the user group corresponding to the user client, it can respond to the data connection request of the user client according to the parameters, and send the requested data to the user client. For example, if it is determined that the user group to which the user belongs is UG_2 according to the user ID, it can determine that the initial RTT in the initial data transmission parameters of the user group is Init_rtt_2, and the initial cwnd is Init_cwnd_2. In this way, when sending data to the user client, the RTT in the initial network parameters is set to Init_rtt_2, and the cwnd in the initial network parameters is set to Init_cwnd_2, and then the corresponding data transmission can be performed.

[0105] Congestion control algorithms in the related art can be divided into two types. One is a "rule-based" congestion control algorithm based on the experience of experts. This type of algorithm judges whether congestion phenomenon will occur in the current network based on packet loss information, time delay information, or bandwidth detection information, and so on, and then performs actions such as congestion avoidance. These methods often rely on the experience of network personnel to set congestion avoidance methods, and lack very good adaptability to network conditions and related businesses (e.g., audio-video business), resulting in relatively little profit from optimizing network transmission performance. The other type of congestion control algorithm is mainly based on the method of artificial intelligence, and such a type of method completely overturns the existing congestion control algorithm or cannot be fully deployed due to many problems in terms of stability, precision control, etc. The self-adaptive congestion control method for user groups proposed in the embodiments of the present application (i.e., the above data transmission control method) can solve the above problem relatively well, and by utilizing network measurement technology, the advantages of conventional congestion control are retained, and new congestion control parameters are set on the basis of conventional congestion control algorithms, which avoids the challenges faced by congestion control algorithms based on machine learning in the related art in terms of convergence, and can ensure the stability of the proposed self-adaptive congestion control method to a certain extent. At the same time, the embodiments of the present application divide client IPs into different user groups based on rules such as regions, operators, and access methods, and set dedicated initial congestion control parameters for each user group by a single reinforcement learning module based on the reinforcement learning method, thereby realizing precise and lightweight control of congestion avoidance in the network traffic transmission process. The self-adaptive congestion control method proposed in the embodiments of the present application can obtain relatively large benefits in terms of setting the initial parameters of existing congestion control algorithms, improve the network transmission performance of video and audio traffic, and is advantageous in improving the competitiveness of audio and video businesses.

[0106] FIG. 9 is another flow diagram of a data transmission control method provided by an embodiment of the present application. As shown in FIG. 9, the flow can be implemented by four modules, including a network measurement module 10, an intelligent learning module 20, a parameter calculation module 30, and a transmission control module 40. Here, the network measurement module 10 can obtain historical network performance data 101 of multiple users, and divide the users into multiple user groups according to IP addresses, obtain user group information 102 of each user, and determine group network performance data corresponding to each user group according to the historical network performance data 101 of each user. Furthermore, the network measurement module 10 can also perform weighting calculation on the group network performance data of multiple user groups to obtain weighted network performance data 103. The intelligent learning module 20 can take the weighted network performance data as a state value, and take the ratio of the effective throughput and the smoothed RTT in the group network performance data as a reward value, and perform reinforcement learning through a reinforcement learning model to obtain a learned data transmission quality index. The parameter calculation module 30 can calculate data transmission parameters 302 corresponding to each user group by adopting the data transmission quality index 301 learned by the reinforcement learning model 201, and form a data transmission parameter list 403. When the transmission control module 40 receives a data transmission request 401 uploaded by a user, it can determine the user group to which the user belongs based on the user's IP address, obtain the user group information 402, and then inquire the data transmission parameters corresponding to the user group in the data transmission parameter list 403, and adopt the corresponding data transmission parameters to perform data transmission control 404 for the data transmission process, thereby improving the transmission performance of the data transmission.

[0107] As can be seen from the above description, the data transmission control method provided by the embodiment of the present application obtains historical network performance data of a plurality of data transmission objects, and calculates group network performance data of each object group (i.e., the above user group) based on the historical network performance data. The object group is a group obtained by dividing a plurality of data transmission objects, performs weighting calculation on the group network performance data to obtain overall network performance data, and calls a trained reinforcement learning model to make prediction based on the overall network performance data and the group network performance data of each object group. Obtains a data transmission quality index of each object group, calculates and obtains data transmission parameters of each object group based on the data transmission quality index and the group network performance data, and when receiving a data transmission request sent from a target data transmission object, determines target data transmission parameters based on the object group to which the target data transmission object belongs, and controls a data transmission process based on the target data transmission parameters.

[0108] Thus, the data transmission control method provided in the embodiment of the present application performs group division on data transmission objects, and then employs the recent period of historical network performance data of each object group to perform reinforcement learning to obtain optimal network data transmission parameters at the current time for each object group. The method performs fine-grained control on the network data transmission parameters, and can match optimal network data transmission parameters at the current time for each data transmission object. In this way, the data transmission performance of the data transmission object can be greatly improved.

[0109] In order to better implement the above data transmission control method, the embodiment of the present application further provides a data transmission control device, which may be integrated in a terminal or a server.

[0110] As shown in Fig. 10, a structural schematic diagram of a data transmission control device provided by an embodiment of the present application. The data transmission control device can include an acquisition unit 1001, a first calculation unit 1002, a learning unit 1003, a second calculation unit 1004, and a control unit 1005. Wherein, the acquisition unit 1001 is configured to acquire historical network performance data of each data transmission object among a plurality of data transmission objects, and calculate group network performance data of each object group among a plurality of object groups according to the historical network performance data, where the plurality of object groups are obtained by dividing the plurality of data transmission objects. The first calculation unit 1002 is configured to perform weighting calculation on the group network performance data of each object group to obtain overall network performance data of the plurality of data transmission objects. The learning unit 1003 is configured to input the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group. The second calculation unit 1004 is configured to determine data transmission parameters of corresponding object groups according to the data transmission quality index and the group network performance data of each of the object groups. The control unit 1005 is configured to determine target data transmission parameters according to the object group to which the target data transmission object belongs when receiving a data transmission request sent from any one of the target data transmission objects among the plurality of data transmission objects, and control a data transmission process corresponding to the data transmission request according to the target data transmission parameters.

[0111] In some embodiments, the learning unit is further configured to calculate reinforcement learning parameters of corresponding object groups based on the group network performance data of each object group, determine the overall network performance data as a state value of the reinforcement learning model, determine the reinforcement learning parameters as a reward value of the reinforcement learning model, input the state value and the reward value into the trained reinforcement learning model, and output a data transmission quality index of each object group by the trained reinforcement learning model.

[0112] In some embodiments, the first calculation unit is further configured to obtain group network performance data of each object group, extract effective throughput data and smoothed round-trip transmission time data in the group network performance data of each object group, calculate a ratio between the effective throughput data and the smoothed round-trip transmission time data corresponding to each object group, and determine the ratio as the reinforcement learning parameter of the corresponding object group.

[0113] In some embodiments, the obtaining unit is further configured to obtain Internet Protocol address data of each data transmission object among the plurality of data transmission objects, obtain historical network performance data of each data transmission object, divide the plurality of data transmission objects into a plurality of object groups based on the Internet Protocol address data of each data transmission object, and calculate group network performance data of the corresponding object group based on the historical network performance data of the data transmission objects in each object group.

[0114] In some embodiments, the second calculation unit is further configured to determine a historical network performance data set corresponding to the corresponding object group based on the historical network performance data of each data transmission object in each object group, calculate an average value of the historical network performance data in each historical network performance data set, and determine the average value corresponding to each historical network performance data set as group network performance data of the corresponding object group.

[0115] In some embodiments, the first calculation unit is further configured to obtain a weighting coefficient for each object group, and perform a weighting calculation on the group network performance data of each object group based on the weighting coefficient, so as to obtain an overall network performance data corresponding to the multiple data transmission objects.

[0116] In some embodiments, the first calculation unit is further configured to obtain a quantity of data transmission objects contained in each object group, and determine a ratio between the quantity of data transmission objects in each object group and the quantity of the plurality of data transmission objects as a weighting coefficient of the corresponding object group.

[0117] In some embodiments, the second calculation unit is further configured to extract effective throughput data and minimum round trip transmission time data of each object group from the above group network performance data of each object group, calculate initial window data of the corresponding object group based on the effective throughput data, minimum round trip transmission time data, and data transmission quality index of each object group, and determine data transmission parameters of the corresponding object group based on the initial window data of each object group.

[0118] In some embodiments, the control unit is further configured, when receiving a data transmission request sent from a target data transmission object of any one of the plurality of data transmission objects, to obtain a target Internet Protocol address of the target data transmission object, determine a target object group to which the target data transmission object belongs based on the target Internet Protocol address, obtain target data transmission parameters of the target object group, and perform data transmission to the target data transmission object based on the target data transmission parameters.

[0119] It is necessary to note that each of the above units can be realized as an independent entity, or can be arbitrarily combined, or can be realized as the same or multiple entities. The specific implementation of each of the above units can be referred to the above method embodiments, and detailed description is omitted here.

[0120] As can be seen from the above description, the data transmission control device provided by the embodiment of the present application uses an acquisition unit 1001 to acquire historical network performance data of each data transmission object among a plurality of data transmission objects, and calculates group network performance data of each object group among a plurality of object groups according to the historical network performance data. A first calculation unit 1002 performs weighting calculation on the group network performance data to obtain overall network performance data of the plurality of data transmission objects. A learning unit 1003 inputs the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group. A second calculation unit 1004 determines data transmission parameters of the corresponding object group according to the data transmission quality index and the group network performance data of each object group. When receiving a data transmission request sent from any one of the target data transmission objects among the plurality of data transmission objects, a control unit 1005 determines target data transmission parameters according to the object group to which the target data transmission object belongs, and controls a data transmission process according to the target data transmission parameters.

[0121] Thus, the data transmission control method provided in the embodiment of the present application performs group division on data transmission objects, and then employs the recent period of historical network performance data of each object group to perform reinforcement learning to obtain optimal network data transmission parameters at the current time for each object group. The data transmission control method performs fine-grained control on the network data transmission parameters, and can match optimal network data transmission parameters at the current time for each data transmission object. In this way, the data transmission performance of the data transmission object can be greatly improved.

[0122] The embodiment of the present application further provides a computer device. The computer device may be a terminal or a server. As shown in FIG. 11, it is a structural schematic diagram of a computer device provided by the embodiment of the present application. The computer device may include components such as a processing unit 1101 of one or more processing cores, a storage unit 1102 of one or more storage media, a power supply module 1103, and an input module 1104. As can be understood by those skilled in the art, the structure of the computer device shown in FIG. 11 does not constitute a limitation on the computer device, and may include more or less components than those shown, or may combine some components or be arranged with different components. Here, the processing unit 1101 is the control center of the computer device, and uses various interfaces and lines to connect each part of the entire computer device, and operates or executes at least one of the software programs and modules stored in the storage unit 1102, and performs various functions of the computer device and processes data by calling up data stored in the storage unit 1102. In some embodiments, the processing unit 1101 may include one or more processing cores, and the processing unit 1101 may integrate an application processor and a modem processor, where the application processor mainly processes an operating system, an object interface, and an application program, etc., and the modem processor mainly processes wireless communication. As can be understood, the modem processor may not be integrated in the processing unit 1101.

[0123] The storage unit 1102 may be configured to store software programs and modules, and the processing unit 1101 may execute various functional applications and data processing by operating the software programs and modules stored in the storage unit 1102. The storage unit 1102 may mainly include a program storage area and a data storage area. Here, the program storage area may store an operating system, an application program required for at least one function (e.g., a voice playback function, an image playback function, and a web page access, etc.), and the data storage area may store newly created data based on the use of the computer device, etc. In addition, the storage unit 1102 may include a high-speed random access memory, and may further include a non-volatile memory, such as at least one magnetic disk storage device, a flash memory device, or other volatile solid-state storage device. The storage unit 1102 may further include a memory controller to provide the processing unit 1101 with access to the storage unit 1102.

[0124] The computer device further includes a power supply module 1103 for supplying power to each component, and the power supply module 1103 is logically connected to the processing unit 1101 through a power management system, so that the power management system can realize functions such as charging, discharging management, and power consumption management. The power supply module 1103 may further include any components such as one or more DC or AC power sources, a recharging system, a power failure detection circuit, a power converter or inverter, and a power status indicator.

[0125] In some embodiments, the computing device may further include an input module 1104 that may be configured to receive input numeric or textual information and generate keyboard, mouse, joystick, optical, or trackball signal inputs for object setting and function control.

[0126] Although not shown here, the computer device may further include a display unit, etc., but detailed description will be omitted. In this embodiment, the processing unit 1101 in the computer device uploads executable instructions corresponding to the processes of one or more application programs in the storage unit 1102 according to the following instructions, and operates the application programs stored in the storage unit 1102 by the processing unit 1101 to realize various functions. For example, the present invention includes a function of obtaining historical network performance data of each data transmission object among the plurality of data transmission objects, and calculating group network performance data of each object group among the plurality of object groups based on the historical network performance data; a function of performing weighting calculation on the group network performance data of each object group to obtain overall network performance data; a function of inputting the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; a function of determining data transmission parameters of a corresponding object group based on the data transmission quality index and the group network performance data of each object group; and a function of determining target data transmission parameters based on the object group to which the target data transmission object belongs when a data transmission request sent from any one of the target data transmission objects among the plurality of data transmission objects is received, and controlling the data transmission process based on the target data transmission parameters.

[0127] As described, the computer device provided by the embodiments of the present application belongs to the same concept as the method in the above embodiments, and the specific implementation of each of the above operations can be referred to the above embodiments, and detailed descriptions are omitted here.

[0128] As can be understood by those skilled in the art, all or some of the steps in the various methods of the above embodiments can be completed by instructions, or can be completed by controlling related hardware by instructions. The instructions can be stored in a computer-readable storage medium, and can be uploaded and executed by a processor.

[0129] Thus, the embodiment of the present application provides a computer-readable storage medium, in which a number of executable instructions are stored, which can be uploaded by a processor to execute any of the steps of the method provided by the embodiment of the present application. For example, the executable instructions can execute the following steps: obtain historical network performance data of each data transmission object among the plurality of data transmission objects, and calculate group network performance data of each object group among the plurality of object groups according to the historical network performance data; perform weighting calculation on the group network performance data of each object group to obtain total network performance data; input the total network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; determine data transmission parameters of the corresponding object group according to the data transmission quality index and the group network performance data of each object group; and when receiving a data transmission request sent from any one target data transmission object among the plurality of data transmission objects, determine target data transmission parameters according to the object group to which the target data transmission object belongs, and control the data transmission process according to the target data transmission parameters.

[0130] For specific implementation of each of the above operations, refer to the above embodiments, and detailed explanations will be omitted here.

[0131] The computer-readable storage medium may include a read only memory (ROM, Read Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk, an optical disk, or the like.

[0132] The executable instructions stored in the computer-readable storage medium can execute the steps of any method provided by the embodiments of the present application. Therefore, the beneficial effects that can be achieved by any method provided by the embodiments of the present application can also be achieved. For details, please refer to the above embodiments, and detailed descriptions will be omitted here.

[0133] According to one aspect of the present invention, a computer program product or a computer program is provided, the computer program product or the computer program includes executable instructions, the executable instructions being stored in a computer-readable storage medium, a processor of a computer device reads the executable instructions from the computer-readable storage medium, and the processor executes the executable instructions to cause the computer device to perform the method provided in various selectable implementation forms of the data transmission control method.

[0134] The above provides a detailed introduction of the data transmission control method, device, computer readable storage medium, computer device, and computer program product provided by the embodiments of the present application. Although the present application uses specific examples to describe the principles and embodiments of the present application, the above examples are only used to help understand the method of the present application and its core ideas. At the same time, those skilled in the art can make modifications to both the specific embodiments and the scope of application based on the ideas of the present application. Therefore, the contents of this specification should not be understood as limitations on the present application.

Claims

1. The method is a data transmission control method executed by a computer device, the method comprising: obtaining historical network performance data for each data transmission object of a plurality of data transmission objects, and calculating group network performance data for each object group of a plurality of object groups based on the historical network performance data, wherein the plurality of object groups are obtained by dividing the plurality of data transmission objects; performing a weighting calculation on the group network performance data of each object group to obtain a comprehensive network performance data of the plurality of data transmission objects; inputting the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; determining data transmission parameters of corresponding object groups based on the data transmission quality index and the group network performance data of each of the object groups; When receiving a data transmission request sent from a target data transmission object among the plurality of data transmission objects, determining target data transmission parameters based on an object group to which the target data transmission object belongs, and controlling a data transmission process corresponding to the data transmission request based on the target data transmission parameters.

2. inputting the overall network performance data and the group network performance data of each object group into a trained reinforcement learning model to obtain a data transmission quality index of each object group; calculating reinforcement learning parameters for the corresponding object group based on the group network performance data of each object group; determining the overall network performance data as state values ​​of the reinforcement learning model and determining the reinforcement learning parameters as reward values ​​of the reinforcement learning model; inputting the state value and the reward value into the trained reinforcement learning model, and outputting a data transmission quality index for each object group by the trained reinforcement learning model.

3. Calculating reinforcement learning parameters for a corresponding object group based on the group network performance data for each object group, obtaining group network performance data for each object group; extracting effective throughput data and smoothed round trip transmission time data from the group network performance data for each object group; calculating a ratio between effective throughput data and smoothed round trip transmission time data corresponding to each object group; and determining the ratio as a reinforcement learning parameter for a corresponding object group.

4. obtaining historical network performance data for each data transmission object of the plurality of data transmission objects and calculating group network performance data for each object group of the plurality of object groups based on the historical network performance data, obtaining Internet Protocol address data for each data transmission object of the plurality of data transmission objects and obtaining historical network performance data for each data transmission object; dividing the plurality of data transmission objects into a plurality of object groups based on the Internet Protocol address data of each data transmission object; and calculating group network performance data for the corresponding object group based on the historical network performance data of the data transmission objects in each object group.

5. Calculating group network performance data of corresponding object groups based on historical network performance data of data transmission objects in each object group, determining a set of historical network performance data corresponding to a corresponding object group based on the historical network performance data of each data transmission object in each object group; calculating an average value of the historical network performance data in each historical network performance data set; and determining an average value corresponding to each historical network performance data set as the group network performance data for the corresponding object group.

6. performing a weighting calculation on the group network performance data of each object group to obtain a comprehensive network performance data of the plurality of data transmission objects; obtaining a weighting factor for each object group; and performing a weighting calculation on the group network performance data of each object group based on the weighting coefficient to obtain an overall network performance data corresponding to the plurality of data transmission objects.

7. The step of obtaining a weighting factor for each object group includes: obtaining the number of data transmission objects contained in each object group; 7. The method of claim 6, further comprising determining a ratio between the number of data-transfer objects in each object group and the number of data-transfer objects in the plurality of object groups as a weighting factor for the corresponding object group.

8. The step of determining data transmission parameters of the corresponding object group based on the data transmission quality index and the group network performance data of each of the object groups includes: extracting effective throughput data and minimum round trip transmission time data for each object group from the group network performance data for each object group; calculating initial window data of a corresponding object group according to the effective throughput data, the minimum round trip time data, and the data transmission quality index of each object group; and determining data transmission parameters of the corresponding object groups based on the initial window data of each object group.

9. a step of determining target data transmission parameters based on an object group to which the target data transmission object belongs when a data transmission request transmitted from any one of the plurality of data transmission objects is received, and controlling a data transmission process corresponding to the data transmission request based on the target data transmission parameters, When receiving a data transmission request sent from any one target data transmission object among the plurality of data transmission objects, obtaining a target Internet Protocol address of the target data transmission object; determining a target object group to which the target data transmission object belongs based on the target Internet Protocol address; obtaining target data transmission parameters of the target object group; and performing a data transmission for the target data transmission object based on the target data transmission parameters.

10. A data transmission control device, the device comprising: an obtaining unit configured to obtain historical network performance data for each data transmission object of a plurality of data transmission objects and calculate group network performance data for each object group of a plurality of object groups based on the historical network performance data, wherein the plurality of object groups are obtained by dividing the plurality of data transmission objects; a first calculation unit configured to perform a weighting calculation on the group network performance data of each object group to obtain an overall network performance data of the plurality of data transmission objects; a learning unit configured to input the overall network performance data and the group network performance data of each of the object groups into a trained reinforcement learning model to obtain a data transmission quality index of each of the object groups; a second calculation unit configured to determine data transmission parameters of corresponding object groups according to the data transmission quality index and the group network performance data of each of the object groups; a control unit configured to, when receiving a data transmission request sent from any one of the plurality of data transmission objects, determine target data transmission parameters based on an object group to which the target data transmission object belongs, and control a data transmission process corresponding to the data transmission request based on the target data transmission parameters.

11. A computer device comprising: a memory; a processor; and executable instructions stored in the memory and operable on the processor, the processor being configured to cause the computer device to implement a data transmission control method according to any one of claims 1 to 9 when executing the executable instructions.

12. A computer program comprising: A computer program configured to cause a computer device to implement the data transmission control method according to any one of claims 1 to 9 when the computer program is executed by a processor of the computer device.

Images