Flow control model training method, devices, computer equipment and storage medium
Patent Information
- Application Number
- HK42023073618
- Authority / Receiving Office
- HK · HK
- Patent Type
- Patents
- Current Assignee / Owner
- Filing Date
- 2023-05-25
- Publication Date
- 2026-09-04
- Estimated Expiration
- 2041-10-17
AI Technical Summary
Existing adaptive bitrate control algorithms, such as the GCC algorithm in WebRTC, suffer from latency and inaccurate flow control in real-time media communication, resulting in untimely network status feedback and affecting the quality of multimedia data transmission.
A flow control model training method is adopted, which uses a pre-trained base model, combined with offline and online training, and uses offline sample data and cumulative reward reference values for rolling training to generate a target flow control model, thereby improving the accuracy of flow control decisions.
By combining offline and online training, the accuracy and adaptability of the flow control model were improved, thereby enhancing the quality of multimedia communication and user experience.
Smart Images

Figure 00000000_0000_ABST
Abstract
Description
[0001] This application is a divisional application filed with the Chinese Patent Office on October 18, 2021, with application number 202111211909.8, entitled "Method and apparatus for flow control of multimedia data and training of flow control model", the entire contents of which are incorporated herein by reference. Technical Field
[0002] This application relates to the field of artificial intelligence technology, and in particular to a flow control model training method, apparatus, computer equipment, and storage medium. Background Technology
[0003] With the development of network technology, more and more multimedia data transmission scenarios require real-time acquisition of network status in order to implement flow control for multimedia data transmission. Taking real-time voice or video calls as an example, these calls are typically implemented using network RTC (Real-Time Communication). In RTC calls, it is usually necessary to monitor the network status in real time and then modify the overall call configuration based on the real-time network status. For example, if the network status is good, the encoding bitrate can be increased. Therefore, how to provide timely feedback on the complex and ever-changing network status in multimedia data transmission is a hot topic.
[0004] In traditional solutions, the adaptive bitrate control algorithm currently used most often is the GCC algorithm in WebRTC, which is a network congestion control algorithm used for real-time media communication. However, the GCC algorithm will produce a certain delay when used in practice, and it relies too much on experience configuration, which will result in inaccurate flow control in actual control. Summary of the Invention
[0005] Therefore, it is necessary to provide a flow control model training method, apparatus, computer equipment, and storage medium that can improve the accuracy of flow control in response to the above-mentioned technical problems.
[0006] A flow control model training method, the method includes:
[0007] Obtain the basic flow control model obtained by pre-training multiple pre-trained sample groups;
[0008] For each offline period in offline training, offline sample data for the next offline period is generated based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data.
[0009] Based on the offline sample data for each offline period, determine the reference value for the cumulative offline reward for each offline period;
[0010] Based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, the basic flow control model is trained in offline periodic rolling training until the offline training stops when the stopping condition is met, thus obtaining the intermediate flow control model.
[0011] For each online period in online training, the reference value of the cumulative online reward for the current online period is determined by using the online sample data of the current online period.
[0012] Based on the online sample data and online cumulative reward reference values corresponding to multiple online cycles, the intermediate flow control model is trained in online cycles until the online training stops when the stopping condition is met, thus obtaining a target flow control model suitable for flow control data prediction in multimedia communication.
[0013] A flow control model training device, the device comprising:
[0014] The acquisition module is used to acquire the basic flow control model obtained by pre-training multiple pre-trained sample groups;
[0015] The generation module is used to generate offline sample data for the next offline period for each offline period during offline training, based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data.
[0016] The first determining module is used to determine the reference value of the offline cumulative reward for each offline period based on the offline sample data for each offline period.
[0017] The offline training module is used to perform offline periodic rolling training on the basic flow control model based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model.
[0018] The second determination module is used to determine the reference value of the cumulative online reward for each online period in the online training by using the online sample data of the current online period.
[0019] The online training module is used to perform online cyclical training on the intermediate flow control model based on the online sample data and online cumulative reward reference value corresponding to multiple online cycles, until the online training stops when the stopping condition is met, so as to obtain the target flow control model suitable for flow control data prediction in multimedia communication.
[0020] A computer device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to perform the following steps:
[0021] Obtain the basic flow control model obtained by pre-training multiple pre-trained sample groups;
[0022] For each offline period in offline training, offline sample data for the next offline period is generated based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data.
[0023] Based on the offline sample data for each offline period, determine the reference value for the cumulative offline reward for each offline period;
[0024] Based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, the basic flow control model is trained in offline periodic rolling training until the offline training stops when the stopping condition is met, thus obtaining the intermediate flow control model.
[0025] For each online period in online training, the reference value of the cumulative online reward for the current online period is determined by using the online sample data of the current online period.
[0026] Based on the online sample data and online cumulative reward reference values corresponding to multiple online cycles, the intermediate flow control model is trained in online cycles until the online training stops when the stopping condition is met, thus obtaining a target flow control model suitable for flow control data prediction in multimedia communication.
[0027] A computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, performs the following steps:
[0028] Obtain the basic flow control model obtained by pre-training multiple pre-trained sample groups;
[0029] For each offline period in offline training, offline sample data for the next offline period is generated based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data.
[0030] Based on the offline sample data for each offline period, determine the reference value for the cumulative offline reward for each offline period;
[0031] Based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, the basic flow control model is trained in offline periodic rolling training until the offline training stops when the stopping condition is met, thus obtaining the intermediate flow control model.
[0032] For each online period in online training, the reference value of the cumulative online reward for the current online period is determined by using the online sample data of the current online period.
[0033] Based on the online sample data and online cumulative reward reference values corresponding to multiple online cycles, the intermediate flow control model is trained in online cycles until the online training stops when the stopping condition is met, thus obtaining a target flow control model suitable for flow control data prediction in multimedia communication.
[0034] A computer program product or computer program includes computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium and executes the computer instructions, causing the computer device to perform the following steps: acquiring a basic flow control model pre-trained from multiple pre-trained sample groups; for each offline period in offline training, generating offline sample data for the next offline period based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication state data; and according to each offline period... Offline sample data is used to determine the offline cumulative reward reference value for each offline period. Based on the offline sample data and offline cumulative reward reference values corresponding to multiple offline periods, the basic flow control model is subjected to offline periodic rolling training until the offline training stopping condition is met, resulting in an intermediate flow control model. For each online period in online training, the online cumulative reward reference value for the current online period is determined using the online sample data of the current online period. Based on the online sample data and online cumulative reward reference values corresponding to multiple online periods, the intermediate flow control model is subjected to online periodic rolling training until the online training stopping condition is met, resulting in a target flow control model suitable for flow control data prediction in multimedia communication processes.
[0035] The aforementioned flow control model training method, apparatus, computer equipment, storage medium, and computer program, through pre-training, enable the basic flow control model to closely resemble the decision-making methods of the previous version of the flow control model. This avoids deploying an untrained flow control model, which could potentially cause user discomfort and degrade the user experience. After pre-training, offline training is performed based on offline sample data and offline cumulative reward reference values, simulating changes in encoded data and communication state data. Offline training allows the flow control model to be trained as much as possible before official deployment, thereby improving the accuracy of flow control decisions. After offline training, online training is finally executed, which perceives real-time states, predicts in real-time, and adaptively improves the corresponding flow control decisions. Because online training continuously updates the flow control model, it continuously improves the predictive performance of the flow control model, greatly enhancing flow control accuracy and thus improving multimedia communication quality and user experience. Attached Figure Description
[0036] Figure 1 This is an application environment diagram of a method for flow control of multimedia data transmission in one embodiment.
[0037] Figure 2 This is a flowchart illustrating a method for flow control of multimedia data in one embodiment;
[0038] Figure 3 This is a schematic diagram of the multi-head attention processing structure in one embodiment;
[0039] Figure 4 This is a schematic diagram of the agent model structure in one embodiment;
[0040] Figure 5 This is a schematic diagram of the interaction principle based on an agent model in one embodiment;
[0041] Figure 6 This is a flowchart illustrating a flow control model training method in one embodiment;
[0042] Figure 7 This is a flowchart illustrating the flow control model training method in another embodiment;
[0043] Figure 8 This is a schematic diagram of the offline training process in one embodiment;
[0044] Figure 9 This is a schematic diagram of the structure of an environment simulator in one embodiment;
[0045] Figure 10 This is a structural block diagram of a device for flow control of multimedia data in one embodiment;
[0046] Figure 11 This is an internal structural diagram of a computer device in one embodiment;
[0047] Figure 12 This is a structural block diagram of a flow control model training device in one embodiment. Detailed Implementation
[0048] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.
[0049] Figure 1 This is an application environment diagram of a method for flow control of multimedia data in one embodiment. (Refer to...) Figure 1This method for flow control of multimedia data is applied to a multimedia flow control system. The multimedia flow control system includes a first terminal 102, a second terminal 104, and a server 106. The first terminal 102, the second terminal 104, and the server 106 can each be used individually to execute the method for flow control of multimedia data provided in this embodiment, or they can be used collaboratively to execute the method for flow control of multimedia data provided in this embodiment. Taking the collaborative execution of the method for flow control of multimedia data provided in this embodiment by the first terminal 102, the second terminal 104, and the server 106 as an example: Both the first terminal 102 and the second terminal 104 are equipped with multimedia communication clients. The first terminal 102 and the second terminal 104 can communicate with each other via the server 106, for example, through audio and video communication. During this process, the server can obtain the current encoding data and communication status data reported by the multimedia communication clients of the first terminal 102 and the second terminal 104. Then, based on the data reported by each terminal, the method for flow control of multimedia data mentioned in the embodiments of this application is executed to predict the target flow control data (which can also be understood as the target flow control strategy) corresponding to the corresponding terminal, and then the target flow control data is applied to the client running on each terminal to perform flow control processing on the multimedia data generated in the next cycle.
[0050] It should be noted that the first terminal and the second terminal mentioned above are only illustrative. In actual use, there may be only one terminal or more than two terminals, etc. This application embodiment does not limit this.
[0051] The server can be a standalone physical server, a server cluster or distributed system consisting of multiple physical servers, or a cloud server providing cloud computing services. The terminal can be a smartphone, tablet, laptop, desktop computer, smart speaker, smartwatch, in-vehicle terminal, smart TV, etc., but is not limited to these. The terminal and server can be directly or indirectly connected via wired or wireless communication, which is not limited herein.
[0052] This application also relates to the field of artificial intelligence (AI). AI is the theory, methods, technology, and application systems that utilize digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to achieve optimal results. In other words, AI is a comprehensive technology within computer science that attempts to understand the essence of intelligence and produce a new type of intelligent machine capable of reacting in a manner similar to human intelligence. AI studies the design principles and implementation methods of various intelligent machines, enabling them to possess perception, reasoning, and decision-making capabilities. Specifically, this application relates to supervised learning, reinforcement learning, attention mechanisms, generative adversarial networks, and multi-task learning within the field of AI.
[0053] Supervised learning is a method of machine learning that learns or builds a pattern (function / model) from training data and uses this pattern to predict new instances. Training data consists of input data and expected outputs. The output of the function can be a continuous value (called regression analysis) or a predicted classification label (called classification).
[0054] Reinforcement Learning (RL) is a field within machine learning that emphasizes how to act based on the environment to maximize expected benefits. It is the third fundamental machine learning method, alongside supervised and unsupervised learning. Unlike supervised learning, reinforcement learning does not require labeled data pairs or precise correction of suboptimal solutions. Its focus is on finding a balance between exploration (of unknowns) and exploitation (of existing knowledge)—the "exploration-exploitation" exchange in reinforcement learning.
[0055] Attention Mechanism: The attention mechanism originates from imitating the process of biological observation, that is, aligning internal experience with external stimuli to increase attention to certain areas.
[0056] Generative Adversarial Networks (GANs): GANs are an unsupervised learning method that learns by having two neural networks compete against each other. A GAN consists of a generator and a discriminator. The generator randomly samples data from the latent space as input, and its output should mimic real samples in the training set as closely as possible. The discriminator takes either real samples or the generator's output as input, and its goal is to distinguish the generator's output from real samples as much as possible. The generator, in turn, tries to deceive the discriminator. The two networks compete against each other, constantly adjusting their parameters, with the ultimate goal of making it impossible for the discriminator to determine whether the generator's output is genuine.
[0057] Multi-task learning: Multiple related tasks are learned together. During the learning process, information from related fields is shared and supplemented through shallow shared representations, thereby promoting mutual progress among multiple tasks.
[0058] Before explaining the method for flow control of multimedia data provided in this application, let's first introduce the concept of flow control:
[0059] In this embodiment, the main focus is on flow control of multimedia data transmission. Flow control can be understood as adjusting the transmission configuration of multimedia data. Specifically, it mainly involves adjusting the encoded data used during multimedia data transmission. For example, taking video as the multimedia data, the flow control process could involve real-time adjustment of encoded data such as resolution during video transmission in a live video call.
[0060] It is understandable that flow control typically occurs periodically, meaning that flow control data for subsequent multimedia data transmission is predicted based on the acquired current communication state data. Therefore, the flow control process is actually periodic. Based on the above description, the flow control process mentioned in this application actually predicts the flow control data to be used for multimedia data transmission in the next cycle based on data from the current cycle, such as communication state data. In the embodiments of this application, the flow control data for the next cycle is used as the target flow control data, which can be used to indicate the encoded data to be used in the next cycle.
[0061] In some embodiments, such as Figure 2 As shown, a method for flow control of multimedia data is provided. The method is illustrated using a computer device as an example. Specifically, the computer device can be... Figure 1 The method for flow control of multimedia data, which involves terminals or servers in the system, includes the following steps:
[0062] Step S202: During multimedia communication, the current encoded data in the current period and the communication status data generated by flow control processing based on the current encoded data are obtained; the current encoded data is determined based on the historical flow control data of the previous period.
[0063] Multimedia communication is a process of transmitting multimedia data, specifically audio and video data. Examples include real-time audio and video communication, and the transmission and playback of audio and video data. Current encoded data refers to the encoded data within the current period. Encoded data are encoding parameters used to guide the multimedia communication client in presenting multimedia data, and may include at least one of the following: bitrate, resolution, or frame rate. Communication status data characterizes the state of multimedia communication within the current communication period, and can be generated by the combined effect of the current network environment and encoded data. Communication status data includes at least one of the following: network packet loss rate, network latency information, network jitter information, or stuttering rate.
[0064] It should be noted that, based on the definition of periodic flow control mentioned above, the "current period" can refer to a time period, specifically defined by two moments. For example, if the current moment is 1:08 PM and the length of the "current period" is 8 minutes, then the other moment used to determine the current period is 1:00 PM, and thus the period from 1:00 PM to 1:08 PM is the "current period." Based on the above, taking a multimedia communication process as a real-time video call as an example, the current encoded data within the current period refers to the video encoded data actually used within the current period. Combining the example of the "current period" above, the current encoded data can refer to the video encoded data used from 1:00 PM to 1:08 PM.
[0065] It is understandable that the communication status data generated by flow control processing using the current encoded data represents the network communication status after the current video encoded data has been used up in the current period. For example, if the current encoded data is determined at 1 PM, then flow control processing can be performed based on the current encoded data during the period from 1 PM to 1:08 PM (the current period). The network status at 1:08 PM after the flow control processing is completed is represented by the "communication status data generated by flow control processing using the current encoded data".
[0066] It should also be noted that the communication status data generated by flow control processing of the current encoded data can be a comprehensive result of the network communication status presented throughout the entire cycle after using the current encoded data, or the status presented at the end of the current cycle as described above. This application does not limit this.
[0067] Regarding the origin of the current encoded data, as can be seen from the above process, "target flow control data can be used to indicate the encoded data used in the next cycle," thus historical flow control data from previous cycles can also be used to indicate the current encoded data used in the current cycle. Similarly, the current encoded data can be determined based on historical flow control data from previous cycles. Specifically, it can be determined based on historical flow control data from the previous cycle.
[0068] It should be noted that multimedia data typically occurs between the terminal and the server, and the server can usually monitor the network status, thus allowing communication status data to be obtained from the server. As for encoded data, it can be directly obtained from the client on the terminal side used for multimedia transmission.
[0069] In some embodiments, taking real-time audio and video communication scenarios as an example, the overall process of real-time audio and video communication is complex, the network environment fluctuates during user calls, and terminal devices are diverse. Therefore, during real-time audio and video communication, the client and server report communication status data and encoded data, and the background flow control logic processes the data and outputs the current flow control decision (i.e., the target flow control data) to ensure the stability of real-time communication as much as possible.
[0070] In this embodiment, to address the issue of excessive and mixed current status reports, indicators crucial for flow control decisions are selected from the various reported status data. Refer to Table 1, which contains the indicator information used in this embodiment:
[0071] Table 1
[0072]
[0073]
[0074] The quality metrics mentioned in the table above can be selectively used in some embodiments of this application. For example, since the audio no-quality reference score and video no-quality reference score can be used as measures of audio or video quality respectively, and the quantization parameter can also be used as a measure of audio or video quality, if the audio no-quality reference score and video no-quality reference score are selected, the quantization parameter can be omitted. Furthermore, since data such as sampling rate and encoding type are usually fixed values in actual implementation, these data can be excluded from selection.
[0075] Step S204: Combine the communication status data and the current encoded data in the current period to obtain the combined data to be processed.
[0076] Specifically, the computer device can combine the communication status data and the current encoded data within the current cycle to obtain the combined data to be processed.
[0077] It is understandable that the content of communication status data is usually numerical. Therefore, even if there is multi-dimensional communication status data, different dimensions of communication status data can be represented by vectors composed of different numerical values. Similarly, the content of encoded data is usually numerical, so encoded data of different dimensions can also be represented by vectors. Therefore, in step S204, "combination processing" can refer to concatenating two data sets to form a whole for easier subsequent processing. It should be noted that combination processing is not limited to concatenation; other further processing can occur before combination, such as dimensionality reduction when there are too many data dimensions, or convolution processing to extract key information. This application embodiment does not specifically limit this.
[0078] In some embodiments, the communication state data and the current encoded data in the current period are combined to obtain the combined data to be processed, including: performing convolution processing on the current encoded data and the communication state data respectively to obtain their respective convolution processing features; and combining the convolution processing features to obtain the combined data to be processed.
[0079] Specifically, the computer device can perform convolution processing on the current encoded data and communication status data separately to obtain their respective convolution processing features. The convolution processing can specifically be one-dimensional convolution processing. Furthermore, the computer device can combine these convolution processing features to obtain the combined data to be processed.
[0080] In some embodiments, a pre-trained flow control model is deployed on the computer device. This model combines the communication state data and the current encoded data within the current period to obtain the combined data to be processed. All indicators in the specific indicator column of Table 1 above have their own numerical characteristics. Since the flow control model is built based on a deep learning network, these indicator data need to be uniformly normalized. The computer device can normalize each indicator data to a value range of 0 to 1 according to its numerical characteristics. That is, after normalizing the current encoded data and the indicator data corresponding to each indicator in the communication state data, they are then combined to obtain the combined data to be processed.
[0081] Since the reported current encoding data and communication status data are time-series data, in order to predict more accurate flow control decisions, the computer equipment can assemble the reported indicator data from a preset number of times into a historical state distribution, and then combine them to obtain the combined data to be processed. For example, the computer equipment can assemble the historical state distribution from the first 8 reported indicator data corresponding to 14 indicator distributions, rather than relying solely on the reported data from the previous moment. After data preprocessing and logical combination of some indicator data, a 14*8 dimension combined data to be processed can be obtained.
[0082] For example, considering a period from 1 PM to 1:08 PM, the communication status data for the current period actually refers to the communication status data at 1:08 PM after the flow control processing during that time. The current encoding data refers to the encoding used during the same period. Therefore, under the above description, both the communication status data and the current encoding data for the current period only have a single value within that period. It is understandable that relying solely on a single value makes it difficult to guarantee the accuracy of the flow control prediction results for the next period.
[0083] In the aforementioned scenario, during actual implementation, the current period can be further divided into multiple sub-periods. For example, the period from 1 PM to 1 PM 8 minutes can be further divided into 8 sub-periods: 1 PM to 1 PM 1 minute, 1 PM 1 minute to 1 PM 2 minutes, 1 PM 2 minutes to 1 PM 3 minutes, ..., 1 PM 7 minutes to 1 PM 8 minutes. Each sub-period can yield communication status data and encoded data. If the communication status data and encoded data comprise 14 types of data, then each type of data in the current period can yield 8 values, thus forming an 8-dimensional vector. Accordingly, in step 204, a one-dimensional convolution can be performed on the 8-dimensional vector corresponding to each type of data.
[0084] It is understood that for any type of data, when performing one-dimensional convolution on the corresponding 8-dimensional vector, the data can undergo one-dimensional convolution once or multiple times; this embodiment does not impose specific limitations on this. Furthermore, one-dimensional convolution can transform data of any dimension by setting the convolution kernel. Thus, the 8-dimensional vector corresponding to each type of data can be transformed into a vector of any dimension. For example, the 8-dimensional vector corresponding to each type of data can be transformed into a 128-dimensional vector.
[0085] Based on the above explanation, if the communication status data and encoded data comprise 14 data types, then the convolutional processing feature corresponding to each data type can be a vector after convolutional processing. Therefore, combining the convolutional processing features corresponding to each data type yields a matrix composed of vectors. For example, if the convolutional processing feature corresponding to each data type is a 128-dimensional vector, then the 14 data types can form a 14*128 matrix. It should be noted that the periods and sub-periods mentioned in the above process are listed as having the same length only for ease of explanation. In actual implementation, the period lengths can be the same or different, and this application does not specifically limit this.
[0086] In the above embodiments, by performing convolution processing on the communication state data and the current encoded data respectively, and then combining them, important features in the current encoded data and the communication state data can be filtered out, thereby improving the accuracy of the prediction results.
[0087] Step S206: Perform self-attention mechanism processing on the combined data to be processed to obtain intermediate processing features, and perform multi-task classification based on the intermediate processing features to output target flow control data of at least one dimension.
[0088] Among them, the self-attention mechanism is a variant of the attention mechanism, which reduces the reliance on external information and is better at capturing the internal correlations of data or features.
[0089] Specifically, the computer device can process the combined data to be processed using a self-attention mechanism to obtain intermediate processing features. Then, based on the intermediate processing features, multi-task classification is performed, and target flow control data with at least one dimension is output.
[0090] In this step, the combined data to be processed is processed using a self-attention mechanism, which mainly involves increasing the weight of certain dimensions of the combined data to be processed, so that these dimensions are more closely related to the output of the subsequent processing.
[0091] Furthermore, as the above process shows, target flow control data can be used to indicate the encoded data to be used in the next cycle. Therefore, outputting target flow control data in at least one dimension can be considered as outputting encoded data in at least one dimension. The dimensions corresponding to encoded data typically include encoding type, resolution, and frame rate, and the values for each dimension are usually explicit and exhaustive. Therefore, given that all values are known, the task of outputting encoded data can be considered as a task of classification based on all known values; this is the origin of the multi-task classification mentioned in this step. Also, in this step, "multi" in "multi-task" corresponds to "at least one dimension." For example, if target flow control data in three dimensions needs to be output, then "multi-task classification" corresponds to three tasks.
[0092] In some embodiments, target flow control data can be used to indicate the encoded data to be used in the next cycle. Therefore, target flow control data can be a number corresponding to the encoded data type, or it can be an identifier corresponding to the encoded data type. This application does not specifically limit the representation of target flow control data. Based on the above description, it can be seen that encoded data can be directly determined through flow control data.
[0093] In some embodiments, a pre-trained flow control model may be deployed on the computer device. This model can combine communication state data and current encoded data within the current period to obtain combined data to be processed. The combined data to be processed is then subjected to a self-attention mechanism to obtain intermediate processing features. Based on these intermediate processing features, multi-task classification is performed, and target flow control data with at least one dimension is output. For details regarding the training process of this flow control model, please refer to the following embodiments.
[0094] Step S208: Based on the target encoded data determined by the target flow control data, trigger flow control processing on the multimedia data generated in the next cycle during the multimedia communication process.
[0095] Specifically, a computer device can trigger flow control processing on multimedia data generated in the next cycle of multimedia communication based on target encoded data determined by target flow control data. When this application is executed by a server, the server can send the target flow control data to the terminal, and the terminal can then determine the target encoded data that matches the target flow control data, thereby performing flow control processing on the multimedia data generated in the next cycle of multimedia communication based on the target encoded data. When this application is executed by a terminal, the terminal can directly determine the target encoded data that matches the target flow control data, thereby performing flow control processing on the multimedia data generated in the next cycle of multimedia communication based on the target encoded data.
[0096] In some embodiments, the target flow control data may specifically include at least one of an encoding bitrate category, a resolution category, or a frame rate category. Correspondingly, the target encoding data may specifically include at least one of an encoding bitrate, a resolution, or a frame rate. That is, when the computer device predicts the encoding bitrate category, the terminal may select one encoding bitrate belonging to that category; when the computer device predicts the resolution category, the terminal may select one resolution belonging to that category; when the computer device predicts the frame rate category, the terminal may select one frame rate belonging to that category. Furthermore, the terminal may apply at least one of the selected encoding bitrate, resolution, or frame rate to the multimedia communication client to perform flow control processing on the multimedia data generated in the next cycle during multimedia communication.
[0097] It should be noted that the method by which the terminal selects a specific encoding bitrate from the encoding bitrate category, a specific resolution from the resolution category, and a specific frame rate from the frame rate category can be random selection, or it can be a selection of an intermediate value, an extreme value, or a corresponding preset value, etc. This application embodiment does not limit this.
[0098] The aforementioned method for flow control of multimedia data combines the current encoded data within the current cycle with the communication status data generated by flow control processing based on the current encoded data. This results in combined data that integrates multiple indicator dimensions. The current encoded data and communication status data can reflect the overall network environment and the quality of the user's subjective experience. Therefore, combining and processing the current encoded data and communication status data provides a more comprehensive guide for flow control data prediction. Next, the combined data is processed using a self-attention mechanism followed by multi-task classification, outputting target flow control data with at least one dimension. This allows for the differentiation of importance among different types of encoded data and communication status data through the self-attention mechanism, fully leveraging the primary and secondary derivation role of high-level representation features. This makes the output target flow control data more adaptable to the complex and ever-changing real-world network environment when used for flow control processing in the next cycle, resulting in better flow control performance. Furthermore, the use of a self-attention mechanism, which captures internal data correlations and reduces reliance on external information, also makes the output flow control data more adaptable to the complex and ever-changing real-world network environment.
[0099] Furthermore, when the execution entity is the terminal, compared to the GCC algorithm, the flow control decision-maker changes from the backend server to the terminal, thus ensuring the timeliness of flow control decisions. Additionally, since the above method can be executed based solely on communication status and encoded data, without relying on other algorithms, it can comprehensively and accurately reflect network changes in multimedia communication scenarios.
[0100] In conjunction with the above embodiments, in some embodiments, the current encoding data includes at least one of encoding bitrate, resolution, or frame rate; the target flow control data includes at least one of encoding bitrate category, resolution category, or frame rate category; and the communication status data includes at least one of network packet loss rate, network latency information, network jitter information, or stuttering rate.
[0101] Here, the encoding bitrate refers to the proportion of useful information in the data stream after the analog signal has been sampled, quantized, and encoded. Resolution refers to the precision of the screen image, which is the total number of pixels displayed on the monitor. Frame rate refers to the frequency or rate at which images, in units of frames, continuously appear on the monitor. As mentioned in the above embodiments, target flow control data can be used to indicate the encoded data to be used in the next cycle. Therefore, in this embodiment, target flow control data may include at least one of encoding bitrate category, resolution category, or frame rate category, indicating the encoded data to be used in the next cycle through the category. The target flow control data can use a one-hot vector to represent the category of encoded data; this embodiment does not specifically limit this.
[0102] Packet loss rate refers to the ratio of lost data packets to sent data packets. It is typically related to packet length and transmission frequency. Network latency refers to the time it takes for a data packet to travel from the sender to the receiver along the network path. Network jitter refers to the time difference between the maximum and minimum latency, used to assess network stability. Lag rate is the ratio of the duration of lag in multimedia communication to the total duration of multimedia communication.
[0103] In the solution provided in the embodiments of this application, since the above-mentioned multiple data can lead to changes in multimedia communication quality and network status, by using the above-mentioned multiple data as the basis for analyzing the network environment and flow control decisions, the output flow control data can be more adaptable to the complex and ever-changing real network environment, the flow control effect is better, and it can be applied to more multimedia communication scenarios.
[0104] In some embodiments, processing the combined data to be processed using a self-attention mechanism to obtain intermediate processing features includes: encoding the combined data to be processed using at least one self-attention module to obtain intermediate processing features; wherein, when there are multiple self-attention modules, the multiple self-attention modules are connected sequentially, and the input data of the first self-attention module is the combined data to be processed, the input data of non-first self-attention modules is the output feature after encoding the previous self-attention module connected to it, and the output feature of the last self-attention module is the intermediate processing feature.
[0105] The self-attention module is a combination of multiple network structures used to encode different types of coded data and communication state data to obtain intermediate processing features. These intermediate processing features can distinguish the importance of different indicator data. The number of self-attention modules can be set according to needs, and this application embodiment does not impose a specific limitation on this. A larger number of self-attention modules can capture the internal correlation of the data as much as possible, reducing the dependence on external information; however, too many modules will increase the computational load. Therefore, in actual implementation, the number of self-attention modules can be three, thereby balancing the effectiveness of the attention mechanism with the computational load.
[0106] In the scheme provided in the embodiments of this application, since different types of encoded data and communication status data can be distinguished by importance through an attention mechanism during the flow control process, the output flow control data is more adaptable to the complex and ever-changing real network environment, and the flow control effect is better.
[0107] In some embodiments, a self-attention module includes a multi-head attention layer and a forward processing layer. For any self-attention module, the encoding processing steps of any self-attention module include: performing multi-head attention processing on the corresponding input data through the multi-head attention layer in the self-attention module to obtain the multi-head attention processing result; and performing forward processing on the multi-head attention processing result through the forward processing layer in the self-attention module to obtain the output features of the corresponding self-attention module.
[0108] The multi-head attention processing mainly involves splitting the input data into multiple heads, determining the weight coefficient matrix for each head, weighting them, and finally integrating the weighted processing results of each head to obtain the multi-head attention processing result of the self-attention module. The forward processing layer can consist of two fully connected layers; however, this embodiment does not specify the number of fully connected layers in the forward processing layer.
[0109] In the scheme provided in the embodiments of this application, the high-dimensional input data can be split into multiple low-dimensional data through the multi-head attention layer, and the weights of each low-dimensional data can be processed separately, thereby preserving the features of multiple dimensions of the data as much as possible and reducing data loss.
[0110] In some embodiments, the multi-head attention layer in the self-attention module is used to perform multi-head attention processing on the corresponding input data to obtain the multi-head attention processing result. Specifically, this includes: performing fully connected processing on the corresponding input data through the multi-head attention layer in the self-attention module to obtain fully connected features; splitting the fully connected features to obtain multiple fully connected sub-features; performing scale-multiplication attention processing on each fully connected sub-feature to obtain multiple multi-head attention sub-features; concatenating the multiple multi-head attention sub-features to obtain multi-head attention features; and performing fully connected processing on the multi-head attention features to obtain the multi-head attention processing result.
[0111] refer to Figure 3 , Figure 3 This is a schematic diagram illustrating the structural principle of multi-head attention processing in one embodiment. For example... Figure 3 As shown, the computer device can decompose the fully connected features into 2^n fully connected sub-features. Then, it performs scale-multiplication attention processing on each fully connected sub-feature to obtain multiple multi-head attention sub-features. The computer device can concatenate these multi-head attention sub-features to obtain the multi-head attention features. Finally, the computer device performs fully connected processing on the multi-head attention features to obtain the multi-head attention processing result.
[0112] In some embodiments, taking 14*128 dimensional fully connected features as an example, the computer device can decompose the fully connected features into 2^n fully connected sub-features, such as four 14*32 dimensional fully connected sub-features. After performing scale-dot-multiplication attention processing on each fully connected sub-feature to obtain multiple multi-head attention sub-features, the computer device can concatenate each multi-head attention sub-feature to obtain the multi-head attention feature. For example, concatenating four 14*32 fully connected sub-features can regenerate 14*128 data, which is the multi-head attention feature. Furthermore, the computer device performs fully connected processing on the multi-head attention feature to obtain the multi-head attention processing result.
[0113] In the above embodiments, since the scale-dot-multiplication attention processing can adaptively distinguish the importance of features of different dimensions, it can give full play to the primary and secondary derivation role of high-level representation features, thereby making the subsequent output flow control data more adaptable to the complex and ever-changing real network environment, and the flow control effect is better.
[0114] In some embodiments, scale-multiplication attention processing is performed on each fully connected sub-feature to obtain multiple multi-head attention sub-features, including: for any fully connected sub-feature, performing matrix multiplication of the corresponding fully connected sub-feature with itself to obtain a matrix multiplication result; performing scale transformation on the matrix multiplication result to obtain a scale transformation result; mapping the scale transformation result to a weight matrix corresponding to the corresponding fully connected sub-feature through a first activation function; and multiplying the weight matrix with the corresponding fully connected sub-feature to obtain a multi-head attention sub-feature corresponding to the corresponding fully connected sub-feature.
[0115] Specifically, for each of the multiple fully connected sub-features, the computer device can perform scale-multiplication attention processing in the same way to obtain multiple multi-head attention sub-features.
[0116] For details on the method of performing scale-dot multiplication attention processing on individual fully connected sub-features, please refer to [link / reference]. Figure 3 ,like Figure 3 As shown, the computer device can perform matrix multiplication between the fully connected sub-feature and itself to obtain the matrix multiplication result. Then, a scaling transformation is applied to the matrix multiplication result to obtain the scaling transformation result. The scaling transformation result is mapped to a weight matrix corresponding to the corresponding fully connected sub-feature using a first activation function (specifically, a softmax normalization function). Finally, the weight matrix is multiplied with the fully connected sub-feature to obtain the multi-head attention sub-feature corresponding to the corresponding fully connected sub-feature.
[0117] For example, taking a fully connected sub-feature matrix with dimensions 14*32, multiplying the fully connected sub-feature by itself (i.e., multiplying the 14*32 feature matrix by a 32*14 feature matrix) yields a 14*14 matrix. The scaling transformation is primarily to ensure that the scores output by the subsequent first activation function are within a suitable range.
[0118] In the above embodiments, since scale-based dot product attention processing can distinguish the importance of features in different dimensions, the subsequent output flow control data can better adapt to the complex and ever-changing real network environment, resulting in better flow control performance.
[0119] In some embodiments, multi-task classification is performed based on intermediate processing features to output target flow control data of at least one dimension, including: performing residual processing on the intermediate processing features for each dimension to obtain the residual processing results corresponding to each dimension; performing fully connected processing on each residual processing result to obtain the corresponding fully connected processing results; and mapping the fully connected processing results to the target flow control data corresponding to the corresponding dimension through a second activation function.
[0120] Specifically, when multiple dimensions exist, the computer device can perform multi-task classification processing in parallel based on intermediate processing features. That is, for each dimension of the task, the computer device can perform residual processing on the intermediate processing features to obtain the residual processing results corresponding to each dimension. Then, each residual processing result is processed by a fully connected layer to obtain the corresponding fully connected processing result. Next, through a second activation function (specifically, the softmax function), the fully connected processing results are mapped to the target flow control data corresponding to the corresponding dimension.
[0121] It should be noted that, as can be seen from the above embodiments, the target flow control data may include at least one of the following: bitrate category, resolution category, or frame rate category. Therefore, the "at least one dimension" mentioned here corresponds to "at least one" in the above content, meaning that at least one type of encoded data of bitrate category, resolution category, or frame rate category can be output, or other types of encoded data may also be used.
[0122] It is understandable that the above mainly involves three processing steps: residual processing, fully connected layer processing, and activation function processing. If these three processing steps are implemented by a multi-head classifier, then in actual implementation, a multi-head classifier can be designed for each dimension of target flow control data that needs to be output. Therefore, in the actual implementation of the embodiments of this application, the number of multi-head classifiers can be consistent with the total dimension of the output data.
[0123] In the above embodiments, since multi-dimensional flow control data can be output, the output flow control data can better adapt to complex and ever-changing real network environments, resulting in better flow control performance and applicability to more multimedia communication scenarios.
[0124] The process described in the above embodiments mainly involves using a model with a self-attention mechanism to implement the internal processing flow of flow control. In actual implementation, this model can be implemented using a reinforcement learning model or other deep learning models; this application does not specifically limit the type of model. For ease of description and understanding, this application uses an agent model in reinforcement learning (i.e., the flow control model mentioned in this application) as an example to illustrate the above process.
[0125] Taking the multimedia communication process mentioned in this method as an example, specifically the real-time audio and video call process, combined with... Figure 4 and Figure 5 The process of the above flow control method will be illustrated with an example. Figure 4 This is a schematic diagram of the agent model structure in one embodiment. Figure 5 This is a schematic diagram of the interaction principle based on an agent model in one embodiment.
[0126] refer to Figure 4 , Figure 4 The leftmost module is the data preprocessing module. This module is mainly used to acquire the current encoded data within the current period, as well as the communication status data generated by flow control processing based on the current encoded data. The communication status data can be obtained from the server, while the current encoded data can be obtained from the multimedia communication client. The communication status data may include at least one of the following: network packet loss rate, network latency information, network jitter information, or stuttering rate. The current encoded data may include at least one of the following: encoding bitrate, resolution, or frame rate.
[0127] Of course, in actual implementation, the communication status data may also specifically include at least one of video packet loss rate, audio packet loss rate, video jitter information, audio jitter information, video stuttering rate, audio stuttering rate, or audio error hiding information. The current encoding data may also specifically include at least one of video encoding bitrate, hardware / software encoding information, or audio encoding bitrate. This application embodiment does not specifically limit the content included in the communication status data and current encoding data. It should be noted that, since data such as sampling rate and encoding type usually use fixed values in actual implementation, these data may not be included in the communication status data and current encoding data.
[0128] If, after processing by the data preprocessing module, 14 types of indicator data can be obtained from the current encoded data and communication status data, these 14 types of indicator data will then be input into the flow control decision AI agent backbone module. Each type of indicator data will be sequentially input into two one-dimensional convolutional layers (Conv_1d). Therefore, the flow control decision AI agent backbone module can have a total of 28 one-dimensional convolutional layers. Of course, in actual implementation, each type of data can also be configured with other numbers of one-dimensional convolutional layers; this embodiment of the invention does not specifically limit this.
[0129] As can be seen from the above embodiments, the current period can be further divided into sub-periods. Therefore, each type of data can be an 8-dimensional vector. After passing through 2 Conv_1d, each 8-dimensional vector can be transformed into a 128-dimensional vector. Thus, after 1-dimensional convolution processing, a 14*128 matrix can actually be obtained.
[0130] After obtaining a 14*128 matrix, this matrix can then be passed through a self-attention module consisting of a multi-head attention layer and a forward processing layer. Figure 4 In this context, "Multi-head attention" refers to the multi-head attention layer, and "Feedforward" refers to the forward processing layer. The dashed box enclosing both represents the self-attention module. Figure 4 As can be seen from the text, the number of self-attention modules can be more than one; in actual implementation, the number of self-attention modules can be three. This application does not specifically limit this number. It should be noted that... Figure 4 The document does not specify the connection relationships between self-attention modules; in practice, these modules can be connected sequentially. For example, if there are three self-attention modules, a 14*128 matrix can be passed through three sequential self-attention modules. By using a sequential connection method, the internal correlations of the data can be captured as much as possible, reducing dependence on external information.
[0131] After processing by the self-attention module, a 14*128 feature matrix is obtained, meaning the backbone module of the flow control decision-making AI agent can output a 14*128 feature matrix. This 14*128 feature matrix can then be input into a multi-head classifier to achieve multi-task classification. For example... Figure 4 The example in the text provides three dimensions of target flow control data: bitrate, resolution, and frame rate. It should be noted that... Figure 4 The example only provides one multi-head classifier. In actual implementation, the number of multi-head classifiers can be consistent with the total dimension of the output data. That is, for each dimension of target flow control data that needs to be output, a multi-head classifier can be designed.
[0132] A multi-head classifier can include a residual block, fully connected layers (FC), and a softmax layer. The residual blocks can consist of two fully connected layers, with residual connections established between the inputs and outputs of these two layers. For example... Figure 4 As shown on the right, "Multi-head Classifier" refers to a multi-head classifier. Multiple multi-head classifiers can be combined to form a classifier like this. Figure 4 The flow control decision prediction module shown.
[0133] The structure of the "Multi-head attention" layer mentioned above can be found in the following reference. Figure 3 . Figure 3 The "input data" is a 14*128 matrix obtained after one-dimensional convolution. This matrix is then processed through a fully connected layer and further split into four 14*32 matrices via multi-channel splitting. Each 14*32 matrix can be processed according to the refinement process indicated by the dotted line on the right. Specifically, for a given 14*32 matrix, due to the self-attention mechanism, it is first copied three times, serving as V, K, and Q respectively. V is multiplied by Q, the result is scaled, and then normalized exponentiation is applied to the scaled result using an activation function to obtain the weight matrix corresponding to the 14*32 matrix. Multiplying this weight matrix by Q yields the feature matrix of the 14*32 matrix after scale-dot-multiplication attention processing.
[0134] Through the above process, each data stream yields a 14*32 feature matrix. Multi-channel concatenation then generates multi-attention features. Finally, a fully connected layer completes the multi-attention processing, resulting in the multi-attention layer's output. After processing by the multi-attention layer, a forward processing layer can further refine the results, completing one self-attention module. Processing through multiple self-attention modules, such as three, yields intermediate processing features.
[0135] The above Figure 3 , Figure 4 The document includes related explanations, primarily focusing on the structure of the behavior prediction network (also known as the Actor network) in the flow control decision-making AI agent. Figure 4As can be seen, the Actor network mainly consists of a flow control decision-making AI agent backbone module and a flow control decision prediction module. The inputs are the communication status provided by the backend server and the encoded data reported by the client. The output is three-dimensional flow control data used for flow control decisions. These flow control decisions affect the client environment, causing changes in various state indicator values and reward values. Since the flow control decisions are all discrete quantized values, the Actor network can be understood as a multi-task "classification" model.
[0136] Understandably, the flow control decision-making AI agent includes not only the Actor network but also a behavior evaluation network (also known as a Critic network). The Critic network mainly consists of the flow control decision-making AI agent's backbone module and a cumulative reward prediction module. Its input is the same as that of the Actor network, and its output is the cumulative reward prediction value. The cumulative reward prediction module can be configured as follows: Figure 4 As shown in the lower right corner, it can include a residual module and a fully connected layer. It should be noted that... Figure 4 It is understood that the Actor network and the Critc network share a common backbone network. However, in actual implementation, it is also possible to choose not to share it. This application does not impose specific limitations on this.
[0137] Based on the above process, the flow control interaction process of the flow control decision-making AI agent can be referenced. Figure 5 The client state is primarily reflected in the encoded data used by the client in the current period, i.e., the current encoded data within the current period. The flow control decision-making AI agent obtains the communication status from the backend server and the client state from the client, and uses both as the basis for flow control decision prediction, which is then applied to the client and thus perceived by the user.
[0138] In the scheme provided in this application embodiment, since different network environments and client states have varying degrees of importance to flow control decisions, the main factors affecting bitrate switching, resolution, and frame rate adjustment are real-time communication states such as audio and video packet loss rate and uplink / downlink latency. Logical inference data, such as stuttering rate and frame rate, are more inclined to assist in guiding flow control decisions. Therefore, for different environmental statistical states, this application embodiment adopts an attention mechanism to adaptively adjust the importance of different data, fully leveraging the primary and secondary derivation role of high-level representation features, thereby making the output flow control decisions more accurate.
[0139] Furthermore, since flow control decisions affecting user subjective experience are not singular, the encoding bitrate, video resolution, and frame rate of audio and video are all directly reflected in the user experience. Therefore, only when these three parameters reach their optimal values at any given moment can a better product experience be provided to the user. This enables AI intelligence to predict appropriate flow control decision values from multiple perspectives. The embodiments of this application employ a multi-task learning strategy to predict the encoding bitrate, resolution, and frame rate separately, thereby maximizing the product experience provided by flow control decisions in different multimedia communication scenarios.
[0140] The above embodiments mainly illustrate the model application process when performing flow control for multimedia data transmission. It is understood that the model also requires an adaptation training process. See also... Figure 6 This application provides a method for training a flow control model. This method can be applied to a terminal or a server, and this application does not specifically limit the type of the executing entity. The terminal can be, but is not limited to, smartphones, tablets, laptops, desktop computers, smart speakers, smartwatches, in-vehicle terminals, and smart TVs. The server can be a standalone physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.
[0141] Taking the application of this method to computer equipment, and the multimedia communication process involved in this method as an example of a real-time video call, the method includes the following steps:
[0142] Step S602: Obtain the basic flow control model obtained by pre-training multiple pre-trained sample groups.
[0143] The base flow control model is obtained by pre-training on multiple pre-trained sample sets. The main purpose of pre-training is to ensure that the base flow control model has a certain flow control performance. Understandably, reinforcement learning-based training is relatively unstable, especially when the Actor and Critic networks of the flow control model initially have inaccurate predictions. Directly performing offline and online training not only slows down the convergence speed of later reinforcement learning training but also affects the performance of the final trained model. Therefore, it is necessary to refine the initial flow control model based on historical reported data in the early stages to obtain a base flow control model with a certain level of effectiveness, also known as the baseline model.
[0144] The pre-training process can employ supervised learning, and this embodiment does not specifically limit the pre-training process. Details regarding pre-training will be provided in later embodiments.
[0145] Step S604: For each offline period in offline training, offline sample data for the next offline period is generated based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline sample encoding data and offline sample communication status data.
[0146] It's important to note that training a flow control model relies on its interaction with the environment. However, in offline mode, the flow control model cannot interact with the online environment in real time. Therefore, an environment simulator can be used to compensate for this deficiency. In other words, during the offline training phase, the environment simulator can generate offline sample data for the next offline period based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model. The offline sample data of the next offline period can then be used as input to the basic flow control model for training, outputting offline flow control prediction data. This continuous interactive loop enables the offline training of the flow control model.
[0147] It should also be noted that, in order to make the flow control model applicable to abnormal network emergencies, the offline sample data generated offline can be randomly processed to simulate abnormal network emergencies and used for the training process of the flow control model.
[0148] In some embodiments, for any offline period, the offline flow control prediction data refers to the flow control data output after inputting the current offline encoded data and the offline communication state data generated by flow control processing of the current offline encoded data into the basic flow control model. The offline sample data can include the current offline encoded data and offline communication state data of the offline period to represent the network state and client state after the end of the offline period. The offline flow control prediction data of the current offline period is actually the flow control decision to be used in the next offline period. Therefore, it can be understood that step S604 is actually simulating the network state and client state at the end of the next offline period after the aforementioned flow control decision to be used is executed, based on the network state and client state before the start of the next offline period; that is, the offline sample data of the next offline period.
[0149] Step S606: Determine the reference value for the offline cumulative reward for each offline period based on the offline sample data for each offline period.
[0150] Specifically, the computer device can sequentially obtain offline sample data for each interaction cycle during the continuous interaction between the behavior prediction network of the flow control model and the environment simulator, until the offline sample data for each offline cycle is obtained. Then, the computer device can determine the offline cumulative reward reference value for each offline cycle based on the offline sample data for each cycle. For a detailed explanation of how to obtain the offline cumulative reward reference value for each offline cycle based on the offline sample data, please refer to the subsequent related implementation sections.
[0151] In this step, as described above, for any offline period, the offline sample data of that period can represent the network state and client state after the offline period ends. The network state and client state after the offline period end can reflect the audio and video call quality of that offline period. For example, if the smoothness of the audio and video calls in that offline period can be determined based on the offline sample data, then the smoothness reflects the audio and video call quality of that offline period. Therefore, this step essentially quantifies the network state and client state of each offline period into an indicator for evaluating the audio and video call quality of that offline period, obtaining a reference value for the offline cumulative reward of each offline period. Subsequently, the computer equipment can use this evaluation indicator to evaluate the offline flow control prediction data of each offline period, enabling offline rolling training of the basic flow control model.
[0152] Step S608: Based on the offline sample data and offline cumulative reward reference value corresponding to each of the multiple offline periods, perform offline periodic rolling training on the basic flow control model until the offline training stops when the stopping condition is met, and obtain the intermediate flow control model.
[0153] Specifically, the computer device can perform offline periodic rolling training on the basic flow control model based on offline sample data and offline cumulative reward reference values corresponding to multiple offline periods, until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model. The offline training stopping condition can be reaching a preset number of training iterations or the training process converging, etc., and this embodiment does not specifically limit this.
[0154] Step S610: For each online period in online training, determine the reference value of the cumulative online reward for the current online period using the online sample data of the current online period.
[0155] Specifically, after the flow control model has converged during the offline training phase, it can be deployed on a multimedia communication server. Within a fixed time period, the intermediate flow control model predicts flow control data based on online sample data (Boakye online encoding data and online communication state data) and applies it to the multimedia communication client, while simultaneously collecting new changes in state. The online sample data collected within the fixed period is used for online training and updates of the intermediate flow control model.
[0156] It's important to note that, unlike offline training, the interactive objects for online flow control model training are no longer environment simulators, but rather online multimedia communication clients and servers. The flow control model is further optimized and updated based on real-time online feedback data. The online training process is similar to the offline training process. During the online period, multiple deployed servers collect online data and use it to train the intermediate flow control model on the server, fine-tuning the model parameters and continuously improving the prediction performance of the intermediate flow control model to obtain the target flow control model.
[0157] Specifically, for each online period in online training, the computer device determines the reference value for the cumulative online reward of the current online period using the online sample data of that period. The online sample data includes Boakye online encoding data and online communication status data.
[0158] Step S612: Based on the online sample data and online cumulative reward reference value corresponding to each of the multiple online cycles, perform online cycle rolling training on the intermediate flow control model until the online training stopping condition is met, and obtain the target flow control model suitable for flow control data prediction in multimedia communication.
[0159] It should be noted that the concept of online training is consistent with that of offline training, as are the methods for calculating reward values and cumulative reward values, and will not be repeated here. The difference is that the online training phase does not require simulating the generation of sample data for the next cycle; instead, the data can be directly obtained online. The stopping conditions for online training can refer to the setting methods for offline training stopping conditions, and this embodiment does not impose specific limitations on them. The online training process allows the flow control model to continuously adapt to the online network environment and constantly update itself, thereby continuously improving the accuracy of flow control decisions.
[0160] In the scheme provided in this application embodiment, pre-training allows the basic flow control model to closely resemble the decision-making method of the previous version of the flow control model. This avoids deploying an untrained flow control model, which could potentially cause user discomfort and reduce the user experience. After pre-training, offline training is employed. This allows the flow control model to be trained as much as possible before official deployment, thereby improving the accuracy of flow control decisions. Furthermore, since offline sample data can be randomly processed to simulate abnormal network emergencies, training the basic flow control model with randomly simulated offline sample data forces it to adapt to these emergencies, thus enabling it to learn self-adjustment. Finally, online training is performed after offline training. Online training continuously updates the flow control model, continuously improving its prediction performance and ultimately enhancing multimedia communication quality and user experience.
[0161] In some embodiments, the basic flow control model includes a behavior prediction network and a behavior evaluation network, which share a coding structure. The behavior prediction network also includes a multi-head classification structure, and the behavior evaluation network also includes a single-task processing structure. The coding structure includes at least one self-attention module connected in sequence, and each self-attention module includes a multi-head attention layer and a forward processing layer.
[0162] In this context, the behavior prediction network corresponds to the Actor network mentioned in the above embodiments, and the behavior evaluation network corresponds to the Critic network. The shared coding structure of the behavior evaluation network corresponds to the backbone network, and the multi-head classification structure corresponds to the multi-head classifier. All concepts mentioned in this application's embodiments can be referenced from the content mentioned in the above embodiments, and will not be repeated here.
[0163] In the scheme provided in the embodiments of this application, since different types of encoded data and communication state data can be distinguished by importance through an attention mechanism during the flow control processing, the output flow control data is more adaptable to complex and ever-changing real network environments, resulting in better flow control performance. Furthermore, since a self-attention mechanism is used, which can capture the internal correlations of data and reduce dependence on external information, the output flow control data is also more adaptable to complex and ever-changing real network environments.
[0164] In some embodiments, the flow control model training method further includes a flow control model pre-training step, referring to... Figure 7 This step specifically includes the following:
[0165] Step S702: Combine the historical communication status data and historical encoding data corresponding to the same historical period in the historical reported data to form historical sample data.
[0166] Historically reported data refers to communication-related data jointly reported by multimedia communication clients and servers during historical periods. For a specific historical period, the historical encoded data refers to the encoded data used in that historical period, while the historical communication status data refers to the communication status data generated after flow control processing using the aforementioned historical encoded data. Computer devices can combine these two types of data corresponding to the same historical period to form the historical sample data for that historical period.
[0167] Step S704: For the current historical period, based on the historical coding data of the next historical period, determine the historical flow control reference data corresponding to the current historical period, and based on the historical sample data of the current historical period, determine the historical cumulative reward reference value corresponding to the current historical period.
[0168] As can be seen from the above embodiments, the target flow control data can be used to indicate the encoded data used in the next cycle. Therefore, it can be understood that for a given historical cycle, the historical flow control reference data used in the next historical cycle is predicted based on the historical sample data of that historical cycle, and the historical flow control reference data used in that historical cycle is associated with the historical encoded data of the next historical cycle.
[0169] Therefore, the computer device can determine the historical flow control reference data corresponding to the current historical period based on the historical coding data of the next historical period. Referring to the explanation of the cumulative reward reference value in the above embodiments, the computer device can also determine the historical cumulative reward reference value corresponding to the current historical period based on the historical sample data of the current historical period. The calculation method for the historical cumulative reward reference value can refer to the calculation method for the offline cumulative reward reference value in the later embodiments.
[0170] Step S706: Take the historical sample data, historical flow control reference data and historical cumulative reward reference value corresponding to the same historical period as a pre-training sample group.
[0171] Step S708: Pre-train the initial flow control model to be trained based on multiple sets of pre-trained sample groups until the pre-training stopping condition is met, and obtain the basic flow control model.
[0172] Specifically, the computer equipment can use the historical sample data, historical flow control reference data, and historical cumulative reward reference value obtained for each historical period as the pre-training sample group corresponding to each historical period. The pre-training process is then completed through step S708.
[0173] Among them, the pre-training method can supervise the learning training method, and the pre-training stopping condition can be the condition of reaching a preset number of training times or the convergence of the training process, etc. This application embodiment does not specifically limit this.
[0174] In the solution provided in the embodiments of this application, since pre-training can make the basic flow control model closely resemble the decision-making method of the previous version of the flow control model, it can avoid putting the flow control model that has not been pre-trained into use, which may cause user discomfort and reduce the user's product experience.
[0175] In some embodiments, the step of pre-training the initial flow control model to be trained based on multiple sets of pre-training sample groups until the pre-training stopping condition is met, to obtain the basic flow control model, specifically includes the following:
[0176] Based on the behavior prediction network in the initial flow control model to be trained, the historical sample data in the pre-training sample group is processed to output historical flow control prediction data. The first cross-entropy loss is determined based on the historical flow control prediction data and historical flow control reference data corresponding to each pre-training sample group. Based on the behavior evaluation network in the initial flow control model to be trained, the historical sample data in the pre-training sample group is processed to output historical cumulative reward prediction values. The first reward loss is determined based on the difference between the historical cumulative reward prediction values and the corresponding historical cumulative reward reference values. A pre-training loss function is constructed based on the first cross-entropy loss and the first reward loss. The initial flow control model is pre-trained using the pre-training loss function until the pre-training stopping condition is met, resulting in the basic flow control model.
[0177] To facilitate understanding, the training process is illustrated below using an example. Before executing the embodiments of this application, a large number of known [state, action, cumulative reward reference value] triplet data can be constructed. Each triplet data corresponds to a set of pre-trained samples mentioned in the above embodiments.
[0178] Taking the t-th historical period as an example, the pre-training sample group corresponding to the t-th historical period can be denoted as [state] t action t The cumulative reward reference value is t. Where, state t The historical sample data corresponding to the t-th historical period, and action t The historical flow control reference data corresponds to the t-th historical period, while the cumulative reward reference value t corresponds to the historical cumulative reward reference value of the t-th historical period.
[0179] As can be seen from the above embodiments, the behavior prediction network in the initial flow control model can be based on the state. t Predict the historical flow control prediction data for the t-th historical period. Since the historical flow control reference data for the t-th historical period is known, a first cross-entropy loss function can be constructed based on the principle of supervised learning to determine the first cross-entropy loss. Taking the historical flow control prediction data as three-dimensional data, namely bitrate, resolution, and frame rate, as an example, the first cross-entropy loss... cls The first cross-entropy loss can be determined by combining the coding rate cross-entropy loss, resolution cross-entropy loss, and frame rate cross-entropy loss. Specifically, it can be obtained by weighted summation.
[0180] In some embodiments, the first cross-entropy loss can be constructed with reference to the following formula (1):
[0181]
[0182] Formula (1) above is the loss function corresponding to the Actor network during pre-training. Here, N represents the total number of pre-training sample groups, and i represents the i-th pre-training sample group. α represents the weight value of the multi-head prediction cross-entropy loss corresponding to the encoding code rate, B represents the total number of categories of the encoding code rate, and j represents the j-th category of the encoding code rate. ij The value of b is related to the coding bitrate in the historical flow control prediction data of the i-th pre-trained sample group. If the coding bitrate in the historical flow control prediction data of the i-th pre-trained sample group corresponds to the j-th category, then b ij The value of b is 1, otherwise b ij The value of ρ is 0. bij The value of is the probability that the coding rate in the historical flow control prediction data of the i-th pre-trained sample group corresponds to the j-th category.
[0183] Similarly, β represents the weight value of the multi-head prediction cross-entropy loss corresponding to the resolution, R represents the total number of categories at the resolution, and k represents the k-th category at the resolution. ik The value of r is related to the resolution value in the historical flow control prediction data of the i-th pre-trained sample group. If the resolution value in the historical flow control prediction data of the i-th pre-trained sample group corresponds exactly to the k-th category, then r ik The value of r is 1, otherwise r ik The value of ρ is 0. rik The value of is the resolution value in the historical flow control prediction data of the i-th pre-trained sample group, which corresponds exactly to the probability of the k-th category.
[0184] γ represents the weight value of the multi-head prediction cross-entropy loss corresponding to the frame rate, F represents the total number of categories for the frame rate, and l represents the l-th category for the resolution. il The value of f is related to the frame rate in the historical flow control prediction data of the i-th pre-trained sample group. If the frame rate in the historical flow control prediction data of the i-th pre-trained sample group corresponds to the l-th category, then f il The value of f is 1, otherwise f il The value of ρ is 0. fil The value of is the probability of the l-th category, which is the frame rate value in the historical flow control prediction data of the i-th pre-trained sample group.
[0185] Using the above formula (1), the first cross-entropy loss corresponding to the pre-trained sample group can be calculated. As can be seen from the above embodiments, the behavior evaluation network, i.e., the Critic network, in the initial flow control model can be based on the state... tThe predicted historical cumulative reward value for the t-th historical period is denoted as the cumulative reward reference value t. Therefore, the first reward loss, i.e., the loss function of the Critic network, can be constructed by calculating the Euclidean distance between the predicted historical cumulative reward value and the historical cumulative reward reference value, as shown in the following formula (2):
[0186]
[0187] In the above formula (2), N represents the total number of pre-training sample groups, t represents the t-th pre-training sample group, and the definitions of other parameters can be found in the above description.
[0188] After determining the first cross-entropy loss and the first reward loss, the computer device can perform a weighted summation operation on the first cross-entropy loss and the first reward loss to construct a pre-training loss function, which can be referred to in the following formula (3):
[0189] loss exp =loss reg +loss cls (3)
[0190] In the above formula (3), loss exp This represents the pre-training loss function.
[0191] Based on formulas (1) to (3) above, the initial flow control model can be pre-trained. The pre-training stopping condition can be the convergence of the pre-training loss function. It should be noted that the above embodiments train the Actor network and the Critic network together, mainly to maintain the connection between the two networks. In actual implementation, the two networks can also be trained separately, i.e., each using its own loss function until the pre-training stopping condition is met. This application embodiment does not specifically limit the pre-training method.
[0192] In the above embodiments, since pre-training enables the basic flow control model to closely resemble the decision-making method of the previous version of the flow control model, it can avoid the discomfort that users may experience when using an untrained flow control model, thereby reducing the user's product experience.
[0193] In some embodiments, offline sample data includes encoding bitrate and stuttering rate. Based on the offline sample data for each offline period, a reference value for the cumulative offline reward for each offline period is determined, including:
[0194] For the current offline period, the clarity is determined based on the encoding bitrate; the smoothness is determined based on the stuttering rate; the smoothness is determined based on the encoding bitrate of the current offline period and the encoding bitrate of the next offline period; the offline reward reference value is calculated based on the clarity, smoothness, smoothness, video quality reference score, and audio quality reference score; and the offline cumulative reward reference value is determined based on the offline cumulative reward reference value of the next offline period and the offline reward reference value of the current offline period.
[0195] As can be seen from the above embodiments, for any offline period, the offline sample data of that offline period may include the offline encoded data used in that offline period, and the offline communication status data generated after the offline period performs flow control processing on the aforementioned encoded data. The offline encoded data may include the encoding bitrate and the stuttering rate. Therefore, in this embodiment, the smoothness, clarity, and fluidity of each offline period can be determined by the encoding bitrate and stuttering rate of each offline period.
[0196] Specifically, video smoothness can be used to indicate the level of stuttering or buffering experienced by a user while watching a video. Stuttering can be categorized into the number of stutters and the duration of each stutter. The fewer the number of stutters and the shorter the duration, the higher the smoothness. Therefore, F... t Let R(F) represent the stuttering rate during the t-th offline period. t ) represents the smoothness of the t-th offline cycle.
[0197] Video clarity can be measured by its bitrate; generally, the higher the bitrate, the higher the clarity. Therefore, bitrate can be used to measure video quality. t Let R(b) represent the coding bitrate of the t-th offline period. t ) represents the clarity of the t-th offline period.
[0198] Video smoothness refers to the perceptible bitrate switching during video playback. Higher smoothness is characterized by fewer bitrate switching occurrences or smaller bitrate fluctuations. Therefore, smoothness can be achieved through (b... t -b t-1 R(b) is the difference in coding bitrate between two adjacent offline cycles, used to measure the fluctuation in coding bitrate switching between adjacent offline cycles. It can be understood that the smaller the difference, the smoother the coding bitrate switching from the (t-1)th offline cycle to the tth offline cycle. Therefore, R(b) can be used to measure the coding bitrate switching fluctuation between adjacent offline cycles. t -b t-1 ) represents the smoothness of the t-th offline period.
[0199] Based on the above explanation of the parameters, the computer device can calculate the offline reward reference value using the following formula (4):
[0200] reward t =λ b R(b t )-λ f R(F t )-λ s R(b t -b t-1 )+λ a R(Qα t )+λ v R(Qv t (4)
[0201] In the above formula (4), reward t Let Qα represent the offline reward reference value for the t-th offline period. t Qv represents the audio quality score without a reference in the t-th offline period. t This indicates that the audio in the t-th offline period has no quality reference score. And λ b Weights representing sharpness, λ f Weights representing smoothness, λ s Weights representing smoothness, λ a The weights and λ representing the audio's unreferenced quality score v This represents the weight of the video's unrepresented quality score.
[0202] R(*) represents a normalization operation, which simplifies calculation by bringing clarity, fluency, smoothness, audio quality-free reference score, and video quality-free reference score to the same standard. The audio quality-free reference score can be used as a measure of audio clarity, and the video quality-free reference score can be used as a measure of video clarity. The video quality-free reference score can be calculated using algorithms such as the Natural Image Quality Assessment Algorithm or the G.1070 standard. The audio quality-free reference score can be calculated using program loudness measurement algorithms or gap algorithms. This application does not specifically limit the specific calculations in this regard.
[0203] It should be noted that the reason for subtracting λ in the above formula (4) is... f R(F t ) and λ s R(b t -b t-1 ), because in the above formula (4), R(F t R(b) is defined as non-fluency (i.e., the opposite of fluency), and R(b) is defined as non-fluency. t -b t-1) is defined as unsharpness (i.e., the opposite of sharpness). In actual implementation, the definitions of each parameter in the above formula (4) can be combined to derive a calculation process different from formula (4). This application embodiment does not specifically limit this.
[0204] It should also be noted that the audio no-quality reference score and the video no-quality reference score can be used as measures of audio or video quality, respectively, and thus participate in the reward calculation. t In addition to calculating audio and video quality-independent reference scores, quantization parameters can also be used as a measure of audio or video quality. In practical implementation, this is applied when calculating rewards. t Optional.
[0205] It is understandable that the computer device can calculate the reward for each offline period using the above formula (4). t The reference value for calculating the offline cumulative reward in the t-th offline period can be found in the following formula (5):
[0206] Cumulative rewards t =reward t +Cumulative rewards t+1 (5)
[0207] For the offline period, in the above formula (5), the cumulative reward t This represents the reference value for the cumulative offline reward in the t-th offline period, while the cumulative reward... t+1 This represents the reference value for the cumulative offline reward in the (t+1)th offline period. t This represents the reference value of the offline reward for the t-th offline period. From formula (5) above, it can be seen that, given the cumulative reward... t+1 and reward t When the value of is selected, the cumulative reward can be calculated. t
[0208] It should be noted that the calculated cumulative reward t At that time, cumulative rewards are required. t+1 In other words, the cumulative reward value is essentially pre-calculated in reverse order. In practice, the computer equipment can perform offline training in rounds. Before conducting the current round of offline training, the reference value for the offline cumulative reward for each offline cycle in the current round can be determined first.
[0209] Specifically, the computer device can retain the behavior prediction network and behavior evaluation network obtained from the previous training round, namely the Exploitation Actor network and the Exploitation Critic network. Based on the behavior prediction network trained in the previous round, the computer device can process the offline sample data for a certain period to obtain the corresponding offline flow control prediction data. Using an environment simulator, based on the offline sample data and offline flow control prediction data of that offline period, it simulates and generates the offline sample data for the next offline period. This process iterates until the offline sample data for each period is obtained.
[0210] It should be noted that the offline cumulative reward reference value is calculated in reverse order. That is, in order to calculate the offline cumulative reward reference value for each offline period, a cutoff offline period can be set for each round of offline training, and the offline sample data at the cutoff offline period is input into the Critic network obtained in the previous round, that is, input into the Exploitation Critic network, so as to output the predicted offline cumulative reward value at the cutoff offline period, which is used as the offline cumulative reward reference value at the cutoff offline period in the current round of offline training.
[0211] The computer device can calculate the offline reward reference value corresponding to each offline period based on the offline sample data corresponding to each period, according to the above formula (4). In this way, if the offline cumulative reward reference value up to the offline period in the current round of offline training is known, and the offline reward reference value of each offline period in the current round of offline training is also known, the offline cumulative reward reference value of each offline period in the current round of offline training can be calculated in reverse order using the above formula (5).
[0212] It is understandable that, in actual implementation, a preset value can be set as the reference value for the cumulative offline reward up to the offline period in the current round of offline training; this application embodiment does not specifically limit this. Thus, after collecting the reference values for the cumulative offline reward corresponding to each offline period required for the current round of offline training, the current round of offline training can be performed.
[0213] It is understandable that the historical cumulative reward reference value mentioned in the above embodiments can also be calculated by referring to the above process. The difference is that the above process requires setting a preset offline end period. However, since the end time of the multimedia communication process is known for historical sample data, it is not necessary to preset the historical end period. Instead, the end time of the multimedia communication process is set as the last historical period, and the historical cumulative reward reference value corresponding to the last historical period is set to 0 or other preset values. Thus, the historical reward reference value corresponding to each historical period can be obtained through formula (4), and the historical cumulative reward reference value corresponding to each historical period can be calculated in reverse order based on formula (5).
[0214] In the scheme provided in the embodiments of this application, since the reward function can comprehensively reflect the clarity, smoothness and fluidity of the video, the output flow control data can better adapt to the complex and ever-changing real network environment, and the flow control effect is better.
[0215] In some embodiments, the basic flow control model includes a behavior prediction network and a behavior evaluation network. Based on offline sample data and offline cumulative reward reference values corresponding to multiple offline periods, the basic flow control model is subjected to offline periodic rolling training until the offline training stopping condition is met, resulting in an intermediate flow control model, including:
[0216] The behavior prediction network for the current offline period processes the offline sample data to obtain the offline flow control prediction data for the current offline period. Based on the offline flow control prediction data for the current offline period, a second cross-entropy loss is determined, and a first offline objective function is determined according to the second cross-entropy loss. The behavior evaluation network for the current offline period processes the offline sample data to obtain the predicted offline cumulative reward value for the current period. Based on the predicted offline cumulative reward value and the reference value of the offline cumulative reward for the current offline period, a second reward loss is determined, and a second offline objective function is constructed based on the second reward loss. The behavior prediction network is trained using the first offline objective function, and the behavior evaluation network is trained using the second offline objective function, until the offline training stopping condition is met, resulting in an intermediate flow control model.
[0217] like Figure 8 As shown, taking the current offline period as the t-th offline period as an example, the computer device will use state... t The input to the Actor network can output the offline flow control prediction data for the t-th offline period. If there are a total of T offline periods, the second cross-entropy loss for all offline periods can be referred to the following formula (6):
[0218]
[0219] In the above formula (6), ρ t Represents the t-th offline period's action. t The probability distribution. It should be noted that, due to action... t In other words, offline flow control prediction data can include various data types, such as resolution, frame rate, and encoding bitrate. Therefore, for each type of data included in the offline flow control prediction data, a second cross-entropy loss can be calculated. That is, in actual implementation, there can be three losses. ce It should be noted that the second cross-entropy loss mainly encourages the predicted value to explore the action space extensively without overemphasizing the probability of a particular type of action.
[0220] After obtaining the second cross-entropy loss, the computer device can determine the first offline objective function. In actual implementation, the loss function of the offline training algorithm can be combined with the second cross-entropy loss to obtain the first offline objective function. The offline training algorithm can be a proximal policy optimization algorithm, a flexible actuation evaluation algorithm, or a dual-delay deep deterministic policy algorithm, etc., and this application embodiment does not specifically limit it.
[0221] like Figure 8 As shown, taking the current offline period as the t-th offline period as an example, the computer device inputs statet into the Critic network and can output the predicted value of the offline cumulative reward for the t-th offline period. By calculating the predicted value of the offline cumulative reward and the reference value of the offline cumulative reward for each offline period, the second reward loss for each offline period can be determined, and thus the second offline objective function can be constructed. The second offline objective function can be referred to as the following formula (7):
[0222]
[0223] In the above formula (7), the cumulative reward t Forecast values and cumulative rewards t The reference values represent the predicted offline cumulative reward and the reference value of the offline cumulative reward for the t-th offline period, respectively.
[0224] In the scheme provided in the embodiments of this application, offline training allows the flow control model to be trained as much as possible before its formal online application, thereby improving the accuracy of flow control decisions. Furthermore, since offline sample data can be randomly processed to simulate abnormal network emergencies, training the basic flow control model with randomly simulated offline sample data forces the model to adapt to these emergencies, thus enabling it to learn self-adjustment.
[0225] In some embodiments, a construction step of a first offline objective function provided by another exemplary embodiment of this application is shown. This step includes the following:
[0226] The offline sample data of the current offline period is processed through the behavior prediction network of the previous round to obtain the offline flow control experience data of the current offline period; the contrast loss is determined based on the offline flow control experience data, the probability distribution of the offline flow control experience data, and the probability distribution of the offline flow control prediction data corresponding to the current offline period; the reinforcement loss is determined based on the contrast loss and the dominance function value; the reinforcement loss is corrected based on the positive and negative reward mechanism to obtain the corrected reinforcement loss; the first offline objective function is constructed based on the second cross-entropy loss and the corrected reinforcement loss.
[0227] Specifically, such as Figure 8 As shown, the computer device inputs the state into the Exploitation Actor network (which is the behavior prediction network obtained from the previous training round), and can output the action. t The empirical value is the offline flow control experience data for the current offline period. The comparison loss can be referenced by the following formula (8):
[0228]
[0229] In the above formula (8), ρ t ρ represents the probability distribution of offline flow control prediction data. et Represents the probability distribution of offline flow control experience data, action et The offline flow control experience data for the t-th offline period can be represented by an on-hot vector.
[0230] After obtaining the contrast loss, the enhancement loss can be determined based on the contrast loss and the dominance function value. See also Figure 8 The advantage function value for each offline period is the difference between the offline cumulative reward reference value and the offline cumulative reward prediction value for each offline period. This application does not specifically limit the method of determining the enhancement loss based on the contrast loss and the advantage function value, including but not limited to: the computer device can calculate a first product between the contrast loss and the advantage function value, calculate a second product between the cutoff value of the contrast loss and the advantage function value, and select the smaller value from the first product and the second product as the enhancement loss. Specifically, the above process can be referred to in the following formula (9):
[0231] loss ppo =min(loss cmp *adv, clip(loss) cmp ,1-eps,1+eps)*adv);(9)
[0232] In the above formula (9), loss cmp Clip(loss) represents the comparison loss. cmp (1-eps, 1+eps) represents the cutoff value for the comparative loss. `clip` represents the cutoff function, and `eps` represents the interval value. This cutoff function indicates the threshold value when the loss... cmp When the value of is within the interval formed by 1-eps to 1+eps, then for the loss cmp Truncate the loss function so that the truncated loss function is optimized. cmp The value is within this range.
[0233] It should be noted that the advantage function mentioned above mainly corresponds to positive and negative reward mechanisms. A negative advantage function value can be interpreted as a negative reward, and vice versa. To further highlight the positive and negative reward mechanism of the advantage function, the computer device can correct the reinforcement loss. The corrected reinforcement loss is denoted as loss. dual Then, by combining the second cross-entropy loss, the first offline objective function can be constructed, as shown in the following formula (10):
[0234] loss actor =loss dual +λ ce *loss ce (10)
[0235] In the above formula (10), λ ce This represents the weighting coefficient of the second cross-entropy loss.
[0236] In the above embodiments, the first offline objective function is constructed by combining the second cross-entropy loss and the corrected reinforcement loss. In this way, within the current offline period, for the Actor network, the convergence objective relies on the cross-entropy loss of the predicted action (three flow control strategies) and the reinforcement loss combining the advantage function, the predicted action and the empirical action, which can significantly improve the training effect of the Actor network.
[0237] In some embodiments, the enhancement loss is corrected based on a positive and negative reward mechanism to obtain a corrected enhancement loss, including: if the current enhancement loss is negative, the larger value between the current enhancement loss and the second reward loss of a preset multiple is used as the corrected enhancement loss; if the current enhancement loss is non-negative, the current enhancement loss is kept unchanged.
[0238] The above process can be referred to in the following formula (11):
[0239]
[0240] In the above formula (11), x represents the loss. ppo 3*loss critic This represents the second reward loss at a preset multiple, loss ppo This indicates that the loss has been amplified.
[0241] In the scheme provided in the embodiments of this application, the training effect is better because the positive and negative reward mechanism of the advantage function can be highlighted by modifying the reinforcement loss.
[0242] It should be noted that the above process mainly refers to the offline training process and various loss functions. For online periodic rolling training, except that the sample data is acquired online rather than through offline simulation, the other processes can be referenced from the offline training process. Specifically, after offline training is completed, the intermediate flow control model obtained from offline training can be deployed to the backend server of the real-time video call. Within a fixed time period, the intermediate flow control model can predict flow control decisions based on real-time network and client states, and apply them to the client, while simultaneously collecting new changes in state. Based on the paired historical data collected within the fixed period, the flow control decision AI agent is continuously trained and updated. By continuously improving the prediction performance of the agent's deployed model, the quality of real-time video calls and the user experience are improved.
[0243] In some embodiments, the offline sample data based on the current offline period and the offline flow control prediction data output by the basic flow control model can be generated by a generator in the environment simulator. Thus, another exemplary embodiment of this application illustrates an environment simulator training step, which includes the following:
[0244] The communication status data and encoding data corresponding to the same training period from the online data collection are used to form environment sample data. For the current training period, the environment flow control data corresponding to the current training period is determined based on the encoding data of the next training period. The environment sample data and environment flow control data of the current training period are processed by the generator in the environment simulator to be trained to generate environment prediction data for the next training period. The environment loss is determined based on the difference between the environment prediction data of each training period and the environment sample data of each training period.
[0245] The generator loss is determined based on the generation simulation degree when generating environment prediction data for each training cycle. The discriminator loss is determined based on the discrimination accuracy of the discriminator in the environment simulator to be trained when discriminating environment prediction data and environment sample data for each training cycle. Based on the environment loss, generator loss, and discriminator loss, the target loss function of the environment simulator to be trained is constructed. The environment simulator to be trained is then iteratively adversarially trained based on the target loss function until the simulation training stops when the stopping condition is met, thus obtaining the environment simulator after training is completed.
[0246] The online data collected in the above process and the historically reported data mentioned in the previous embodiments are both historical data, and their content may be the same or different. This application embodiment does not specifically limit this. For the current training period t, the communication status data of the t-th training period can be denoted as action. t The environmental flow control data in the t-th training period can be denoted as action. t The environment prediction data for the (t+1)th training cycle, i.e., the next training cycle, can be denoted as state′. t+1 .
[0247] Therefore, the computer device can refer to the following formula (12) to determine the difference between the environmental prediction data for each historical period and the environmental sample data for each historical period:
[0248]
[0249] In the above formula (12), loss mse Let N represent the total number of historical periods, i represent the i-th historical period, y represent the environmental sample data of the i-th historical period, and y′ represent the environmental prediction data of the i-th historical period.
[0250] In addition, the generator loss can be referred to by the following formula (13), and the discriminator loss can be referred to by the following formula (14):
[0251]
[0252]
[0253] Understandably, for formula (14), the training objective is primarily to ensure that the discriminator can identify the generated environment prediction data as fake and the real environment sample data as genuine. For formula (13), the training objective is primarily to ensure that the generator can generate environment prediction data that the discriminator can identify as genuine. It should be noted that the network structures of both the generator and the discriminator in the environment simulator can be based on the Transformer encoder-decoder, the difference being that their outputs are inconsistent. The output of the decoder in the generator is the same as the output of state′. t+1 Data in the same dimensional space, with the output of the discriminator's decoding end passing through two fully connected layers and then an outer softmax layer, can perform binary classification to distinguish between true and false data. For details on the structure of the environment simulator, please refer to [reference needed]. Figure 9 .
[0254] In the solution provided in the embodiments of this application, since an environment simulator can be built based on the structure of a generative adversarial network to generate offline sample data that may cause network fluctuations at any time, the flow control model can be forced to adapt to abnormal network emergencies and learn to adjust itself to improve the flow control effect.
[0255] In one specific embodiment, a flow control model training method is provided, comprising the following steps:
[0256] Historical communication status data and historical coding data corresponding to the same historical period in the historical reported data are combined to form historical sample data; for the current historical period, historical flow control reference data corresponding to the current historical period are determined based on the historical coding data of the next historical period of the current historical period.
[0257] For the current offline period, the clarity is determined based on the encoding bitrate; the smoothness is determined based on the stuttering rate; the smoothness is determined based on the encoding bitrate of the current offline period and the encoding bitrate of the next offline period; the offline reward reference value is calculated based on the clarity, smoothness, smoothness, video quality reference score, and audio quality reference score; and the offline cumulative reward reference value is determined based on the offline cumulative reward reference value of the next offline period and the offline reward reference value of the current offline period.
[0258] Historical sample data, historical flow control reference data, and historical cumulative reward reference values corresponding to the same historical period are used as a pre-training sample group. Based on the behavior prediction network in the initial flow control model to be trained, the historical sample data in the pre-training sample group is processed to output historical flow control prediction data. According to the historical flow control prediction data and historical flow control reference data corresponding to each pre-training sample group, the first cross-entropy loss is determined. Based on the behavior evaluation network in the initial flow control model to be trained, the historical sample data in the pre-training sample group is processed to output historical cumulative reward prediction values. According to the difference between the historical cumulative reward prediction values and the corresponding historical cumulative reward reference values, the first reward loss is determined. Based on the first cross-entropy loss and the first reward loss, a pre-training loss function is constructed.
[0259] The initial flow control model is pre-trained using a pre-training loss function until the pre-training stopping condition is met, thus obtaining the basic flow control model. Communication status data and encoding data corresponding to the same training period from the online collected data are combined to form environmental sample data. For the current training period, the environmental flow control data corresponding to the current training period is determined based on the encoding data of the next training period.
[0260] The generator in the environment simulator to be trained processes the environmental sample data and environmental flow control data of the current training period to generate the environmental prediction data for the next training period. The environmental loss is determined based on the difference between the environmental prediction data and the environmental sample data of each training period. The generator loss is determined based on the generation simulation degree when generating the environmental prediction data for each training period. The discriminator loss is determined based on the discrimination accuracy of the discriminator in the environment simulator to be trained when discriminating between the environmental prediction data and the environmental sample data of each training period. Based on the environmental loss, generator loss, and discriminator loss, a target loss function for the environment simulator to be trained is constructed. Iterative adversarial training is then performed on the environment simulator to be trained based on the target loss function until the simulation training stops, resulting in a completed environment simulator.
[0261] For each offline period in offline training, offline sample data for the next offline period is generated using an environment simulator based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model. The offline sample data includes offline encoding data and offline communication state data. Based on the offline sample data of each offline period, the offline cumulative reward reference value for each offline period is determined. The offline sample data of the current offline period is processed through the behavior prediction network of the current offline period to obtain the offline flow control prediction data of the current offline period. The second cross-entropy loss is determined based on the offline flow control prediction data of the current offline period.
[0262] The offline sample data of the current offline period is processed through the behavior prediction network of the previous offline period to obtain the offline flow control experience data of the current offline period. Based on the offline flow control experience data, the probability distribution of the offline flow control experience data, and the probability distribution of the offline flow control prediction data corresponding to the current offline period, the contrast loss is determined. The reinforcement loss is determined according to the contrast loss and the dominance function value. The dominance function value is the difference between the offline cumulative reward reference value and the offline cumulative reward prediction value of each offline period. If the current reinforcement loss is negative, the larger value between the current reinforcement loss and the second reward loss with a preset multiple is used as the corrected reinforcement loss. If the current reinforcement loss is non-negative, the current reinforcement loss remains unchanged. The first offline objective function is constructed according to the second cross-entropy loss and the corrected reinforcement loss.
[0263] The offline sample data of the current offline period is processed through the behavior evaluation network of the current offline period to obtain the predicted value of the offline cumulative reward of the current offline period. Based on the predicted value of the offline cumulative reward of the current offline period and the reference value of the offline cumulative reward of the current offline period, a second reward loss is determined, and a second offline objective function is constructed based on the second reward loss. The behavior prediction network is trained through the first offline objective function, and the behavior evaluation network is trained through the second offline objective function until the offline training stopping condition is met, thus obtaining the intermediate flow control model.
[0264] For each online period in the online training, the online cumulative reward reference value for the current online period is determined using the online sample data of the current online period. Based on the online sample data and online cumulative reward reference values corresponding to multiple online periods, the intermediate flow control model is subjected to online periodic rolling training until the online training stopping condition is met, thus obtaining a target flow control model suitable for flow control data prediction in multimedia communication.
[0265] In the aforementioned flow control model training method, pre-training allows the basic flow control model to closely resemble the decision-making process of the previous version. This avoids deploying an untrained flow control model, which could potentially cause user discomfort and degrade the user experience. After pre-training, offline training is employed. This allows the flow control model to be trained as extensively as possible before official deployment, improving the accuracy of flow control decisions. Furthermore, by randomly processing offline sample data to simulate abnormal network emergencies, training the basic flow control model with randomly simulated offline sample data forces it to adapt to these situations, enabling it to learn self-adjustment. Finally, online training is performed after offline training. Online training continuously updates the flow control model, continuously improving its prediction performance and ultimately enhancing multimedia communication quality and user experience.
[0266] Furthermore, because the flow control process can differentiate the importance of different types of encoded data and communication state data through an attention mechanism, the output flow control data is more adaptable to complex and ever-changing real-world network environments, resulting in better flow control performance. Additionally, the use of a self-attention mechanism, which captures internal data correlations and reduces reliance on external information, also makes the output flow control data more adaptable to complex and ever-changing real-world network environments.
[0267] Furthermore, since the reinforcement loss can be modified to highlight the positive and negative reward mechanisms of the advantage function, the training effect is better.
[0268] Finally, since the reward function can comprehensively reflect the clarity, smoothness, and fluidity of the video, the output flow control data is more adaptable to the complex and ever-changing real network environment, resulting in better flow control performance.
[0269] Because the flow control model can be trained through a triple training approach—pre-training, offline training, and online training—the accuracy of its predictions can be improved. Furthermore, the flow control process distinguishes the importance of different types of encoded data and communication state data using an attention mechanism, making the output flow control data more adaptable to complex and ever-changing real-world network environments, resulting in better flow control performance. Additionally, the use of a self-attention mechanism, which captures internal correlations within the data and reduces reliance on external information, further enhances the adaptability of the output flow control data to complex and ever-changing real-world network environments.
[0270] In real-time audio or video call scenarios, users may be in various geographical environments, network types, and terminal types when making audio or video calls using everyday terminal devices. These environments may include plains or mountains, indoors or outdoors, and cities or rural areas. Furthermore, users may be on various network types such as wired networks, Wi-Fi, 5G, and 4G, and on different mobile phone models. Naturally, users desire a high QoE (Quality of Experience) during audio and video calls, such as high-definition and smooth audio and video playback. Therefore, in conjunction with real-time audio and video call scenarios, this application also provides a method for flow control of video call data. Taking the application of this method to a computer device as an example, the computer device can specifically be... Figure 1 The method, which involves a terminal or server in the system, includes the following steps:
[0271] During a video call, the computer device acquires the frame rate, encoding bitrate, resolution, encoding type, video no-quality reference score, and audio no-quality reference score for the current period, and uses these as the current encoding data. The computer device also acquires the packet loss rate, network latency information, network jitter information, and frame rate generated by flow control processing based on the current encoding data, and uses these as communication status data. The current encoding data is determined based on the encoding bitrate, resolution, and frame rate of the previous period.
[0272] The computer equipment combines the communication status data and the current encoded data within the current cycle to obtain the combined data to be processed.
[0273] The computer device processes the combined data to be processed using a self-attention mechanism to obtain intermediate processing features. Based on these features, it performs multi-task classification, outputs the encoding bitrate, resolution, and frame rate for the next cycle, and triggers flow control processing on the video call data generated in the next cycle during the real-time video call.
[0274] It is understandable that, in addition to real-time audio and video call scenarios, there can be other multimedia communication scenarios, such as live streaming and real-time audio and video on-demand scenarios. Taking the real-time audio on-demand scenario as an example, this application embodiment also provides a method for flow control of audio on-demand data. The method is described using a computer device as an example, wherein the computer device can specifically be... Figure 1 The method, which involves a terminal or server in the system, includes the following steps:
[0275] During audio playback, the computer device acquires the audio frame rate, audio encoding bitrate, audio encoding type, and audio quality reference score for the current period, and uses these as the current encoding data. The computer device also acquires the packet loss rate, network latency information, network jitter information, and frame rate generated by flow control processing based on the current encoding data, and uses these as communication status data. The current encoding data is determined based on the audio frame rate and audio encoding bitrate of the previous period.
[0276] The computer device combines the communication status data and the current encoded data within the current period to obtain the combined data to be processed;
[0277] The computer device processes the combined data to be processed using a self-attention mechanism to obtain intermediate processing features. Based on these features, it performs multi-task classification, outputs the audio frame rate and audio encoding bitrate for the next cycle, and triggers flow control processing on the audio on-demand data generated in the next cycle during the real-time audio on-demand call.
[0278] It should be understood that, although Figure 2 , Figure 6 The steps in the flowchart are shown sequentially as indicated by the arrows, but these steps are not necessarily executed in the order indicated by the arrows. Unless otherwise specified herein, there is no strict order in which these steps are executed, and they can be performed in other orders. Figure 2 , Figure 6 At least some of the steps in the process may include multiple steps or multiple stages. These steps or stages are not necessarily completed at the same time, but may be executed at different times. The execution order of these steps or stages is not necessarily sequential, but may be executed in turn or alternately with other steps or at least some of the steps or stages in other steps.
[0279] In some embodiments, such as Figure 10 As shown, a device 1000 for flow control of multimedia data is provided. This device can be a software module, a hardware module, or a combination of both integrated into a computer device. Specifically, the device includes: an acquisition module 1002, a combined processing module 1004, a self-attention mechanism processing module 1006, a multi-task classification module 1008, and a flow control processing module 1010, wherein:
[0280] The acquisition module 1002 is used to acquire the current encoded data in the current period and the communication status data generated by flow control processing based on the current encoded data during multimedia communication; the current encoded data is determined based on the historical flow control data of the previous period.
[0281] The combination processing module 1004 is used to combine the communication status data and the current encoded data in the current period to obtain the combined data to be processed;
[0282] The self-attention mechanism processing module 1006 is used to process the combined data to be processed using the self-attention mechanism to obtain intermediate processing features.
[0283] The multi-task classification module 1008 is used to perform multi-task classification based on intermediate processing features and output target flow control data with at least one dimension.
[0284] The flow control processing module 1010 is used to trigger flow control processing on the multimedia data generated in the next cycle during the multimedia communication process based on the target encoded data determined by the target flow control data.
[0285] In some embodiments, the combination processing module 1004 is used to perform convolution processing on the current encoded data and the communication status data respectively to obtain their respective convolution processing features; and combine the convolution processing features to obtain the combined data to be processed.
[0286] In some embodiments, the self-attention mechanism processing module 1006 is used to encode the combined data to be processed through at least one self-attention module to obtain intermediate processing features; wherein, when there are multiple self-attention modules, the multiple self-attention modules are connected in sequence, and the input data of the first self-attention module is the combined data to be processed, the input data of the non-first self-attention modules is the output feature after encoding the previous self-attention module connected to it, and the output feature of the last self-attention module is the intermediate processing feature.
[0287] In some embodiments, the self-attention module includes a multi-head attention layer and a forward processing layer; the self-attention mechanism processing module 1006 includes:
[0288] The multi-head attention processing unit is used to perform multi-head attention processing on the corresponding input data through the multi-head attention layer in the self-attention module to obtain the multi-head attention processing result;
[0289] The forward processing unit is used to perform forward processing on the multi-head attention processing results through the forward processing layer in the self-attention module to obtain the output features of the corresponding self-attention module.
[0290] In some embodiments, the multi-head attention processing unit is specifically used to perform fully connected processing on the corresponding input data through the multi-head attention layer in the self-attention module to obtain fully connected features; to split the fully connected features to obtain multiple fully connected sub-features; to perform scale-multiplication attention processing on each fully connected sub-feature to obtain multiple multi-head attention sub-features; to concatenate the multiple multi-head attention sub-features to obtain multi-head attention features; and to perform fully connected processing on the multi-head attention features to obtain the multi-head attention processing result.
[0291] In some embodiments, the multi-head attention processing unit is further configured to, for any fully connected sub-feature, perform matrix multiplication of the corresponding fully connected sub-feature with itself to obtain a matrix multiplication result; perform scaling transformation on the matrix multiplication result to obtain a scaling transformation result; map the scaling transformation result to a weight matrix corresponding to the corresponding fully connected sub-feature through a first activation function; and multiply the weight matrix with the corresponding fully connected sub-feature to obtain a multi-head attention sub-feature corresponding to the corresponding fully connected sub-feature.
[0292] In some embodiments, the multi-task classification module 1008 is used to perform residual processing on the intermediate processing features for each of at least one dimension to obtain the residual processing results corresponding to each dimension; to perform fully connected processing on each residual processing result to obtain the corresponding fully connected processing results; and to map the fully connected processing results to the target flow control data corresponding to the corresponding dimension through a second activation function.
[0293] For specific limitations regarding the device for flow control of multimedia data, please refer to the limitations on the method for flow control of multimedia data mentioned above, which will not be repeated here. Each module in the aforementioned device for flow control of multimedia data can be implemented entirely or partially through software, hardware, or a combination thereof. These modules can be embedded in hardware or independently of the processor in the computer device, or stored in software in the memory of the computer device, so that the processor can call and execute the corresponding operations of each module.
[0294] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above-described method embodiments.
[0295] In some embodiments, such as Figure 11As shown, a flow control model training device 1100 is provided. This device can be a software module, a hardware module, or a combination of both as part of a computer device. Specifically, the device includes: an acquisition module 1102, a generation module 1104, a first determination module 1106, an offline training module 1108, a second determination module 1110, and an online training module 1112, wherein:
[0296] The acquisition module 1102 is used to acquire the basic flow control model obtained by pre-training multiple pre-trained sample groups;
[0297] The generation module 1104 is used to generate offline sample data for the next offline period for each offline period in offline training, based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data.
[0298] The first determining module 1106 is used to determine the offline cumulative reward reference value for each offline period based on the offline sample data for each offline period.
[0299] The offline training module 1108 is used to perform offline periodic rolling training on the basic flow control model based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model.
[0300] The second determining module 1110 is used to determine the reference value of the online cumulative reward for each online period in the online training by using the online sample data of the current online period.
[0301] The online training module 1112 is used to perform online cyclic rolling training on the intermediate flow control model based on the online sample data and online cumulative reward reference value corresponding to multiple online cycles, until the online training stops when the stopping condition is met, so as to obtain the target flow control model suitable for flow control data prediction in multimedia communication.
[0302] In some embodiments, the basic flow control model includes a behavior prediction network and a behavior evaluation network, which share a coding structure. The behavior prediction network also includes a multi-head classification structure, and the behavior evaluation network also includes a single-task processing structure. The coding structure includes at least one self-attention module connected in sequence, and each self-attention module includes a multi-head attention layer and a forward processing layer.
[0303] In some embodiments, the basic flow control model is obtained through pre-training, and the device further includes a pre-training module; the pre-training module includes:
[0304] The combination unit is used to combine historical communication status data and historical encoded data corresponding to the same historical period in the historical reported data to form historical sample data.
[0305] The first determining unit is used to determine, for the current historical period, the historical flow control reference data corresponding to the current historical period based on the historical coding data of the next historical period of the current historical period;
[0306] The second determining unit is used to determine the historical cumulative reward reference value corresponding to the current historical period based on the historical sample data of the current historical period.
[0307] The third determining unit is used to take historical sample data, historical flow control reference data and historical cumulative reward reference values corresponding to the same historical period as a pre-training sample group;
[0308] The pre-training unit is used to pre-train the initial flow control model to be trained based on multiple sets of pre-training sample groups until the pre-training stopping condition is met, thus obtaining the basic flow control model.
[0309] In some embodiments, the pre-training unit is configured to process historical sample data in the pre-training sample group based on the behavior prediction network in the initial flow control model to be trained, and output historical flow control prediction data; determine a first cross-entropy loss based on the historical flow control prediction data and historical flow control reference data corresponding to each pre-training sample group; process historical sample data in the pre-training sample group based on the behavior evaluation network in the initial flow control model to be trained, and output historical cumulative reward prediction values; determine a first reward loss based on the difference between the historical cumulative reward prediction values and the corresponding historical cumulative reward reference values; construct a pre-training loss function based on the first cross-entropy loss and the first reward loss; and pre-train the initial flow control model using the pre-training loss function until the pre-training stopping condition is met, thereby obtaining the basic flow control model.
[0310] In some embodiments, the first determining module 1206 is configured to, for the current offline period, determine the clarity of the current offline period based on the encoding bitrate of the current offline period; determine the smoothness of the current offline period based on the stuttering rate of the current offline period; determine the smoothness of the current offline period based on the encoding bitrate of the current offline period and the encoding bitrate of the next offline period; calculate the offline reward reference value of the current offline period based on the clarity, smoothness, smoothness, video quality-free reference score, and audio quality-free reference score of the current offline period; and determine the offline cumulative reward reference value of the current offline period based on the offline cumulative reward reference value of the next offline period and the offline reward reference value of the current offline period.
[0311] In some embodiments, the basic flow control model includes a behavior prediction network and a behavior evaluation network; the offline training module 1108 includes:
[0312] The first processing unit is used to process the offline sample data of the current offline period through the behavior prediction network of the current offline period to obtain the offline flow control prediction data of the current offline period.
[0313] The fourth determining unit is used to determine the second cross-entropy loss based on the offline flow control prediction data of the current offline period;
[0314] The fifth determining unit is used to determine the first offline objective function based on the second cross-entropy loss;
[0315] The second processing unit is used to process the offline sample data of the current offline period through the behavior evaluation network of the current offline period to obtain the predicted value of the offline cumulative reward of the current period.
[0316] The sixth determining unit is used to determine the second reward loss based on the predicted value of the offline cumulative reward for the current offline period and the reference value of the offline cumulative reward for the current offline period;
[0317] Construction unit, used to construct a second offline objective function based on the second reward loss;
[0318] The offline training unit is used to train the behavior prediction network using the first offline objective function and the behavior evaluation network using the second offline objective function, until the offline training stops when the stopping condition is met, thus obtaining the intermediate flow control model.
[0319] In some embodiments, the offline training module 1108 further includes:
[0320] The third processing unit is used to process the offline sample data of the current offline period through the behavior prediction network of the previous round to obtain the offline flow control experience data of the current offline period.
[0321] The seventh determining unit is used to determine the contrast loss based on the offline flow control experience data corresponding to the current offline cycle, the probability distribution of the offline flow control experience data, and the probability distribution of the offline flow control prediction data.
[0322] The eighth determining unit is used to determine the enhancement loss based on the contrast loss and the advantage function value; the advantage function value is the difference between the offline cumulative reward reference value and the offline cumulative reward predicted value for each offline period.
[0323] The correction unit is used to correct the enhancement loss based on the positive and negative reward mechanism, so as to obtain the corrected enhancement loss.
[0324] Accordingly, the fifth determining unit is used to construct the first offline objective function based on the second cross-entropy loss and the corrected reinforcement loss.
[0325] In some embodiments, the correction unit is configured to, when the current enhancement loss is negative, use the larger of the current enhancement loss and the second reward loss of a preset multiple as the corrected enhancement loss; if the current enhancement loss is non-negative, keep the current enhancement loss unchanged.
[0326] In some embodiments, the step of generating offline sample data for the next offline period based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model is implemented by a generator in the environment simulator; correspondingly, the device also includes an environment simulator training module.
[0327] The environment simulator training module is used to assemble environment sample data from online collected data of communication status data and encoded data corresponding to the same training period. For the current training period, based on the encoded data of the next training period, the environment flow control data corresponding to the current training period is determined. The generator in the environment simulator to be trained processes the environment sample data and environment flow control data of the current training period to generate environment prediction data for the next training period. The environment loss is determined based on the difference between the environment prediction data and the environment sample data of each training period. The generator loss is determined based on the generation simulation degree when generating the environment prediction data of each training period. The discriminator loss is determined based on the discriminator in the environment simulator to be trained, based on the discrimination accuracy when discriminating the environment prediction data and environment sample data of each training period. Based on the environment loss, generator loss, and discriminator loss, the target loss function of the environment simulator to be trained is constructed. The environment simulator to be trained is then iteratively trained based on the target loss function until the simulation training stops when the stopping condition is met, resulting in the environment simulator after training is completed.
[0328] In one embodiment, a computer device is provided, which may be a terminal or a server, and its internal structure diagram may be as follows: Figure 12As shown, the computer device includes a processor, memory, and a communication interface connected via a system bus. The processor provides computing and control capabilities. The memory includes a non-volatile storage medium and internal memory. The non-volatile storage medium stores the operating system and computer programs. The internal memory provides an environment for the operation of the operating system and computer programs stored in the non-volatile storage medium. The communication interface is used for wired or wireless communication with external terminals; wireless communication can be achieved through Wi-Fi, carrier networks, NFC (Near Field Communication), or other technologies. When executed by the processor, the computer program implements a method for flow control of multimedia data.
[0329] Those skilled in the art will understand that Figure 12 The structure shown is merely a block diagram of a portion of the structure related to the present application and does not constitute a limitation on the computer device to which the present application is applied. Specific computer devices may include more or fewer components than those shown in the figure, or combine certain components, or have different component arrangements.
[0330] In some embodiments, a computer device is also provided, including a memory and a processor, wherein the memory stores a computer program, and the processor executes the computer program to implement the steps in the above method embodiments.
[0331] In some embodiments, a computer-readable storage medium is provided storing a computer program that, when executed by a processor, implements the steps in the above method embodiments.
[0332] In some embodiments, a computer program product or computer program is provided, the computer program product or computer program including computer instructions stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and executes the computer instructions, causing the computer device to perform the steps in the above-described method embodiments.
[0333] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The computer program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the methods described above. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include at least one of non-volatile and volatile memory. Non-volatile memory can include read-only memory (ROM), magnetic tape, floppy disk, flash memory, or optical storage, etc. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM can be in various forms, such as static random access memory (SRAM) or dynamic random access memory (DRAM), etc.
[0334] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.
[0335] The embodiments described above are merely illustrative of several implementation methods of this application, and while the descriptions are relatively specific and detailed, they should not be construed as limiting the scope of the invention patent. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of this application, and these all fall within the protection scope of this application. Therefore, the protection scope of this patent application should be determined by the appended claims.
Claims
1. A method for training a flow control model, characterized in that, The method includes: Obtain the pre-trained basic flow control model; wherein the pre-training steps include: forming historical sample data from historical communication state data and historical coding data corresponding to the same historical period in the historical reported data; for the current historical period, determining the historical flow control reference data corresponding to the current historical period based on the historical coding data of the next historical period, and determining the historical cumulative reward reference value corresponding to the current historical period based on the historical sample data of the current historical period; taking the historical sample data, historical flow control reference data, and historical cumulative reward reference value corresponding to the same historical period as a pre-training sample group; and pre-training the initial flow control model to be trained based on multiple pre-training sample groups; For each offline period in offline training, offline sample data for the next offline period is generated based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data. Based on the offline sample data for each offline period, determine the reference value for the cumulative offline reward for each offline period; Based on the offline sample data and offline cumulative reward reference value corresponding to each of the multiple offline periods, the basic flow control model is subjected to offline periodic rolling training until the offline training stops when the stopping condition is met, thus obtaining the intermediate flow control model. For each online period in online training, the reference value of the cumulative online reward for the current online period is determined by using the online sample data of the current online period. Based on the online sample data and online cumulative reward reference values corresponding to multiple online cycles, the intermediate flow control model is subjected to online cycle rolling training until the online training stops when the stopping condition is met, thus obtaining a target flow control model suitable for flow control data prediction in multimedia communication.
2. The method according to claim 1, characterized in that, The basic flow control model includes a behavior prediction network and a behavior evaluation network. The behavior prediction network and the behavior evaluation network share a coding structure. The behavior prediction network also includes a multi-head classification structure, and the behavior evaluation network also includes a single-task processing structure. The coding structure includes at least one self-attention module connected in sequence. Each self-attention module includes a multi-head attention layer and a forward processing layer.
3. The method according to claim 1, characterized in that, The pre-training of the initial flow control model to be trained based on multiple sets of pre-training sample groups includes: The initial flow control model to be trained is pre-trained using multiple sets of pre-trained sample groups until the pre-training stopping condition is met, thus obtaining the basic flow control model.
4. The method according to claim 3, characterized in that, The process involves pre-training the initial flow control model to be trained using multiple sets of pre-trained sample groups until the pre-training stopping condition is met, resulting in a basic flow control model, including: Based on the behavior prediction network in the initial flow control model to be trained, the historical sample data in the pre-trained sample group is processed to output historical flow control prediction data. The first cross-entropy loss is determined based on the historical flow control prediction data and historical flow control reference data corresponding to each pre-trained sample group. Based on the behavior evaluation network in the initial flow control model to be trained, the historical sample data in the pre-trained sample group is processed to output the historical cumulative reward prediction value. The first reward loss is determined based on the difference between the predicted historical cumulative reward value and the corresponding historical cumulative reward reference value; Based on the first cross-entropy loss and the first reward loss, a pre-training loss function is constructed; The initial flow control model is pre-trained using the pre-training loss function until the pre-training stopping condition is met, thus obtaining the basic flow control model.
5. The method according to claim 1, characterized in that, The offline sample data includes encoding bitrate and stuttering rate. The step of determining the offline cumulative reward reference value for each offline period based on the offline sample data for each offline period includes: For the current offline period, the resolution of the current offline period is determined based on the encoding bitrate of the current offline period; The smoothness of the current offline period is determined based on the stuttering rate of the current offline period. The smoothness of the current offline period is determined based on the coding bitrate of the current offline period and the coding bitrate of the next offline period. Based on the clarity, smoothness, fluidity, video quality-free reference score, and audio quality-free reference score of the current offline period, calculate the offline reward reference value for the current offline period; The offline cumulative reward reference value for the current offline period is determined based on the offline cumulative reward reference value for the next offline period and the offline reward reference value for the current offline period.
6. The method according to claim 5, characterized in that, The step of calculating the offline reward reference value for the current offline period based on the clarity, smoothness, fluidity, video quality-free reference score, and audio quality-free reference score of the current offline period includes: The clarity, smoothness, fluidity, video quality reference score, and audio quality reference score of the current offline period are normalized respectively. The offline reward reference value for the current offline period is obtained by weighting the normalization results.
7. The method according to claim 1, characterized in that, The basic flow control model includes a behavior prediction network and a behavior evaluation network. Based on offline sample data and offline cumulative reward reference values corresponding to multiple offline periods, the basic flow control model undergoes offline periodic rolling training until the offline training stops when the stopping condition is met, resulting in an intermediate flow control model, including: The offline sample data of the current offline period is processed by the behavior prediction network of the current offline period to obtain the offline flow control prediction data of the current offline period. The second cross-entropy loss is determined based on the offline flow control prediction data of the current offline period, and the first offline objective function is determined based on the second cross-entropy loss; The offline sample data of the current offline period is processed by the behavior evaluation network of the current offline period to obtain the predicted value of the offline cumulative reward for the current period. Based on the predicted value of the offline cumulative reward for the current offline period and the reference value of the offline cumulative reward for the current offline period, a second reward loss is determined, and a second offline objective function is constructed based on the second reward loss. The behavior prediction network is trained using the first offline objective function, and the behavior evaluation network is trained using the second offline objective function until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model.
8. The method according to claim 7, characterized in that, The method further includes: The offline sample data of the current offline period is processed by the behavior prediction network of the previous round to obtain the offline flow control experience data of the current offline period. Based on the offline flow control experience data corresponding to the current offline cycle, the probability distribution of the offline flow control experience data, and the probability distribution of the offline flow control prediction data, the contrast loss is determined; The enhancement loss is determined based on the comparative loss and the advantage function value; the advantage function value is the difference between the offline cumulative reward reference value and the offline cumulative reward predicted value for each offline period. Based on the positive and negative reward mechanism, the reinforcement loss is corrected to obtain the corrected reinforcement loss; The step of determining the first offline objective function based on the second cross-entropy loss includes: Based on the second cross-entropy loss and the corrected reinforcement loss, a first offline objective function is constructed.
9. The method according to claim 8, characterized in that, The step of determining the enhancement loss based on the contrast loss and the dominance function value includes: Calculate the first product between the contrast loss and the advantage function value; Calculate the second product between the cutoff value of the contrast loss and the dominance function value, and select the smaller value from the first product and the second product as the enhancement loss; the cutoff value refers to the value formed when the contrast loss is limited to a preset interval.
10. The method according to claim 8, characterized in that, The reinforcement loss is corrected based on a positive and negative reward mechanism to obtain a corrected reinforcement loss, including: If the current enhancement loss is negative, the larger value between the current enhancement loss and the second reward loss with a preset multiple will be used as the corrected enhancement loss. If the current enhancement loss is non-negative, then the current enhancement loss remains unchanged.
11. The method according to any one of claims 1 to 10, characterized in that, The step of generating offline sample data for the next offline period based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model is implemented by a generator in the environment simulator. The training steps of the environment simulator include: The communication status data and encoding data corresponding to the same training period from the online data collection are combined to form environmental sample data; For the current training period, based on the encoded data of the next training period, determine the environmental flow control data corresponding to the current training period; The generator in the environment simulator to be trained processes the environment sample data and environment flow control data of the current training period to generate environment prediction data for the next training period. The environmental loss is determined based on the difference between the environmental prediction data and the environmental sample data for each training cycle. The generator loss is determined based on the generation simulation degree when the generator generates environment prediction data for each training cycle, and the discriminator loss is determined based on the discrimination accuracy when the discriminator in the environment simulator to be trained discriminates environment prediction data and environment sample data for each training cycle. Based on the aforementioned environment loss, generator loss, and discriminator loss, a target loss function for the environment simulator to be trained is constructed. Then, based on the target loss function, iterative adversarial training is performed on the environment simulator to be trained until the simulation training stops when the stopping condition is met, thus obtaining the environment simulator after training is completed.
12. A flow control model training device, characterized in that, The device includes: The acquisition module is used to acquire the pre-trained basic flow control model. The pre-training steps include: forming historical sample data from historical communication state data and historical encoding data corresponding to the same historical period in the historical reported data; for the current historical period, determining historical flow control reference data corresponding to the current historical period based on the historical encoding data of the next historical period, and determining the historical cumulative reward reference value corresponding to the current historical period based on the historical sample data of the current historical period; using the historical sample data, historical flow control reference data, and historical cumulative reward reference value corresponding to the same historical period as a pre-training sample group; and pre-training the initial flow control model to be trained based on multiple pre-training sample groups. The generation module is used to generate offline sample data for the next offline period for each offline period during offline training, based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model; the offline sample data includes offline encoded data and offline communication status data. The first determining module is used to determine the reference value of the offline cumulative reward for each offline period based on the offline sample data for each offline period. The offline training module is used to perform offline periodic rolling training on the basic flow control model based on the offline sample data and offline cumulative reward reference value corresponding to multiple offline periods, until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model. The second determination module is used to determine the reference value of the cumulative online reward for each online period in the online training by using the online sample data of the current online period. The online training module is used to perform online cyclical rolling training on the intermediate flow control model based on the online sample data and online cumulative reward reference value corresponding to multiple online cycles, until the online training stops when the stopping condition is met, so as to obtain a target flow control model suitable for flow control data prediction in multimedia communication.
13. The flow control model training device according to claim 12, characterized in that, The basic flow control model includes a behavior prediction network and a behavior evaluation network. The behavior prediction network and the behavior evaluation network share a coding structure. The behavior prediction network also includes a multi-head classification structure, and the behavior evaluation network also includes a single-task processing structure. The coding structure includes at least one self-attention module connected in sequence. Each self-attention module includes a multi-head attention layer and a forward processing layer.
14. The flow control model training device according to claim 12, characterized in that, The device also includes a pre-training module, which is used to pre-train the initial flow control model to be trained based on multiple sets of pre-training sample groups until the pre-training stopping condition is met, thereby obtaining the basic flow control model.
15. The flow control model training device according to claim 14, characterized in that, The pre-training module is used for: Based on the behavior prediction network in the initial flow control model to be trained, the historical sample data in the pre-trained sample group is processed to output historical flow control prediction data. The first cross-entropy loss is determined based on the historical flow control prediction data and historical flow control reference data corresponding to each pre-trained sample group. Based on the behavior evaluation network in the initial flow control model to be trained, the historical sample data in the pre-trained sample group is processed to output the historical cumulative reward prediction value. The first reward loss is determined based on the difference between the predicted historical cumulative reward value and the corresponding historical cumulative reward reference value; Based on the first cross-entropy loss and the first reward loss, a pre-training loss function is constructed; The initial flow control model is pre-trained using the pre-training loss function until the pre-training stopping condition is met, thus obtaining the basic flow control model.
16. The flow control model training device according to claim 12, characterized in that, The offline sample data includes encoding bitrate and stuttering rate, and the first determining module is used for: For the current offline period, the resolution of the current offline period is determined based on the encoding bitrate of the current offline period; The smoothness of the current offline period is determined based on the stuttering rate of the current offline period. The smoothness of the current offline period is determined based on the coding bitrate of the current offline period and the coding bitrate of the next offline period. Based on the clarity, smoothness, fluidity, video quality-free reference score, and audio quality-free reference score of the current offline period, calculate the offline reward reference value for the current offline period; The offline cumulative reward reference value for the current offline period is determined based on the offline cumulative reward reference value for the next offline period and the offline reward reference value for the current offline period.
17. The flow control model training device according to claim 16, characterized in that, The first determining module is further configured to: The clarity, smoothness, fluidity, video quality reference score, and audio quality reference score of the current offline period are normalized respectively. The offline reward reference value for the current offline period is obtained by weighting the normalization results.
18. The flow control model training device according to claim 12, characterized in that, The basic flow control model includes a behavior prediction network and a behavior evaluation network, and the offline training module is used for: The offline sample data of the current offline period is processed by the behavior prediction network of the current offline period to obtain the offline flow control prediction data of the current offline period. The second cross-entropy loss is determined based on the offline flow control prediction data of the current offline period, and the first offline objective function is determined based on the second cross-entropy loss; The offline sample data of the current offline period is processed by the behavior evaluation network of the current offline period to obtain the predicted value of the offline cumulative reward for the current period. Based on the predicted value of the offline cumulative reward for the current offline period and the reference value of the offline cumulative reward for the current offline period, a second reward loss is determined, and a second offline objective function is constructed based on the second reward loss. The behavior prediction network is trained using the first offline objective function, and the behavior evaluation network is trained using the second offline objective function until the offline training stops when the stopping condition is met, thus obtaining an intermediate flow control model.
19. The flow control model training device according to claim 18, characterized in that, The offline training module is also used for: The offline sample data of the current offline period is processed by the behavior prediction network of the previous round to obtain the offline flow control experience data of the current offline period. Based on the offline flow control experience data corresponding to the current offline cycle, the probability distribution of the offline flow control experience data, and the probability distribution of the offline flow control prediction data, the contrast loss is determined; The enhancement loss is determined based on the comparative loss and the advantage function value; the advantage function value is the difference between the offline cumulative reward reference value and the offline cumulative reward predicted value for each offline period. Based on the positive and negative reward mechanism, the reinforcement loss is corrected to obtain the corrected reinforcement loss; Based on the second cross-entropy loss and the corrected reinforcement loss, a first offline objective function is constructed.
20. The flow control model training device according to claim 19, characterized in that, The offline training module is also used for: Calculate the first product between the contrast loss and the advantage function value; Calculate the second product between the cutoff value of the contrast loss and the dominance function value, and select the smaller value from the first product and the second product as the enhancement loss; the cutoff value refers to the value formed when the contrast loss is limited to a preset interval.
21. The flow control model training device according to claim 19, characterized in that, The offline training module is also used for: If the current enhancement loss is negative, the larger value between the current enhancement loss and the second reward loss with a preset multiple will be used as the corrected enhancement loss. If the current enhancement loss is non-negative, then the current enhancement loss remains unchanged.
22. The flow control model training apparatus according to any one of claims 12 to 21, characterized in that, The step of generating offline sample data for the next offline period based on the offline sample data of the current offline period and the offline flow control prediction data output by the basic flow control model is implemented by a generator in the environment simulator. The device also includes an environment simulator training module, which is used for: The communication status data and encoding data corresponding to the same training period from the online data collection are combined to form environmental sample data; For the current training period, based on the encoded data of the next training period, determine the environmental flow control data corresponding to the current training period; The generator in the environment simulator to be trained processes the environment sample data and environment flow control data of the current training period to generate environment prediction data for the next training period. The environmental loss is determined based on the difference between the environmental prediction data and the environmental sample data for each training cycle. The generator loss is determined based on the generation simulation degree when the generator generates environment prediction data for each training cycle, and the discriminator loss is determined based on the discrimination accuracy when the discriminator in the environment simulator to be trained discriminates environment prediction data and environment sample data for each training cycle. Based on the aforementioned environment loss, generator loss, and discriminator loss, a target loss function for the environment simulator to be trained is constructed. Then, based on the target loss function, iterative adversarial training is performed on the environment simulator to be trained until the simulation training stops when the stopping condition is met, thus obtaining the environment simulator after training is completed.
23. A computer device comprising a memory and a processor, wherein the memory stores a computer program, characterized in that, When the processor executes the computer program, it implements the steps of the method according to any one of claims 1 to 11.
24. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.
25. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the steps of the method according to any one of claims 1 to 11.
