Data flow control method, system and device, electronic equipment and storage medium

Abstract

The disclosure provides a data flow control method, system, device, electronic equipment and storage medium, and relates to the technical fields of computer vision, voice technology, intelligent media and video flow. The specific implementation scheme is: receiving parameter information sent by a first terminal; wherein the parameter information comprises live broadcast parameters and hardware parameters; determining initial encoding parameters based on the live broadcast parameters; determining target encoding parameters based on the hardware parameters and the initial encoding parameters; and sending the target encoding parameters to the first terminal; wherein the target encoding parameters are used for encoding original multimedia data flow to obtain encoded multimedia data flow. The above method can ensure the quality of the multimedia data flow uploaded by the terminal.

Description

Technical Field

[0001] This disclosure relates to the field of computer technology, and in particular to the fields of computer vision, speech technology, intelligent media and video streaming technology. Background Technology

[0002] With the development of technology, users have increasingly higher requirements for the clarity and smoothness of audio and video (especially in live streaming scenarios). However, how to ensure the quality of audio and / or video data streams uploaded by the live streaming client has become a problem that needs to be solved. Summary of the Invention

[0003] This disclosure provides a data flow control method, system, apparatus, electronic device, and storage medium.

[0004] According to a first aspect of this disclosure, a data flow control method is provided, comprising:

[0005] Receive parameter information sent by the first terminal; wherein the parameter information includes live streaming parameters and hardware parameters;

[0006] Based on the live streaming parameters, determine the initial encoding parameters;

[0007] Based on the hardware parameters and the initial encoding parameters, the target encoding parameters are determined;

[0008] The target encoding parameters are sent to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream.

[0009] According to a second aspect of this disclosure, a data flow control system is provided, comprising:

[0010] A data stream control device is configured to receive parameter information sent by a first terminal; wherein the parameter information includes live streaming parameters and hardware parameters; determine initial encoding parameters based on the live streaming parameters; determine target encoding parameters based on the hardware parameters and the initial encoding parameters; and send the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain an encoded multimedia data stream.

[0011] The first terminal is used to send the parameter information and receive the target encoded parameters.

[0012] According to a third aspect of this disclosure, a data flow control device is provided, comprising:

[0013] A receiving module is used to receive parameter information sent by a first terminal; wherein the parameter information includes live streaming parameters and hardware parameters;

[0014] The processing module is used to determine initial encoding parameters based on the live streaming parameters; and to determine target encoding parameters based on the hardware parameters and the initial encoding parameters.

[0015] The sending module is used to send the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream.

[0016] According to a fourth aspect of this disclosure, an electronic device is provided, comprising:

[0017] At least one processor; and

[0018] The memory is communicatively connected to the at least one processor; wherein,

[0019] The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the data flow control method of the first aspect described above.

[0020] According to a fifth aspect of this disclosure, a non-transitory computer-readable storage medium is provided storing computer instructions for causing the computer to perform the aforementioned method.

[0021] According to a sixth aspect of this disclosure, a computer program product is provided, including a computer program that, when executed by a processor, implements the aforementioned method.

[0022] It should be understood that the description in this section is not intended to identify key or essential features of the embodiments of this disclosure, nor is it intended to limit the scope of this disclosure. Other features of this disclosure will become readily apparent from the following description.

[0023] The solution provided in this embodiment can determine the target encoding parameters corresponding to the first terminal by combining the live streaming parameters and hardware parameters of the first terminal. In this way, encoding parameters that are more in line with the requirements of different terminals can be determined, so that each terminal can encode the multimedia data stream based on the encoding parameters that meet its own requirements when conducting live streaming, thus ensuring the quality of the multimedia data stream uploaded by the terminal. Attached Figure Description

[0024] The accompanying drawings are provided to better understand this solution and do not constitute a limitation of this disclosure. Wherein:

[0025] Figure 1 This is a schematic flowchart of a data flow control method according to an embodiment of the present disclosure;

[0026] Figure 2This is a schematic diagram of the composition structure of a data flow control system according to an embodiment of the present disclosure;

[0027] Figure 3 This is a schematic diagram of the composition structure of a data flow control system according to yet another embodiment of the present disclosure;

[0028] Figure 4 This is a schematic diagram of an exemplary component architecture of a data flow control system according to an embodiment of the present disclosure;

[0029] Figure 5 This is a schematic flowchart of a data flow control method according to an embodiment of the present disclosure;

[0030] Figure 6 This is a schematic diagram of the composition structure of a data flow control system according to another embodiment of the present disclosure;

[0031] Figure 7 This is another schematic flowchart of a data flow control method according to an embodiment of the present disclosure;

[0032] Figure 8 This is a schematic diagram of the composition structure of a data flow control device according to an embodiment of the present disclosure;

[0033] Figure 9 This is a schematic diagram of the composition structure of a data flow control device according to another embodiment of the present disclosure;

[0034] Figure 10 This is a block diagram of an electronic device used to implement the data flow control method of the embodiments of this disclosure. Detailed Implementation

[0035] The exemplary embodiments of this disclosure are described below with reference to the accompanying drawings, including various details of the embodiments to aid understanding, and should be considered merely exemplary. Therefore, those skilled in the art will recognize that various changes and modifications can be made to the embodiments described herein without departing from the scope and spirit of this disclosure. Similarly, for clarity and brevity, descriptions of well-known functions and structures are omitted in the following description.

[0036] The first aspect of this disclosure provides a data flow control method, such as... Figure 1 As shown, it includes:

[0037] S101: Receive parameter information sent by the first terminal; wherein, the parameter information includes live streaming parameters and hardware parameters;

[0038] S102: Determine the initial encoding parameters based on the live streaming parameters;

[0039] S103: Determine the target encoding parameters based on the hardware parameters and the initial encoding parameters;

[0040] S104: Send the target encoding parameters to the first terminal; wherein, the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream.

[0041] The data flow control method provided in the first aspect embodiment above can be applied to an electronic device; the electronic device can be a server.

[0042] By adopting the above scheme, the target encoding parameters corresponding to the first terminal can be determined by combining the live streaming parameters and hardware parameters of the first terminal. In this way, encoding parameters that are more in line with the requirements of different terminals can be determined, so that each terminal can encode the multimedia data stream based on the encoding parameters that meet its own requirements when conducting live streaming, thus ensuring the quality of the multimedia data stream uploaded by the terminal.

[0043] In some possible implementations, determining the initial encoding parameters based on the live streaming parameters may include: inputting the live streaming parameters into the target model to obtain the initial encoding parameters output by the target model.

[0044] The live streaming parameters may include at least one of the following: live streaming type and device type. The live streaming type may include at least one of the following: outdoor, live show, gaming, education, finance, sports, etc. The device type may include one of the following: smart terminal, desktop computer, third-party server, etc.

[0045] It should be understood that the aforementioned live streaming parameters are merely illustrative examples. In actual processing, any parameters related to live streaming can be included, such as at least one of the following: live streaming room identifier, streamer information, and broadcast time. This section will not exhaustively list all possible contents that live streaming parameters may include.

[0046] The aforementioned initial encoding parameters may include at least one of the following: initial encoding algorithm, initial encoding method, initial bitrate, initial resolution, and initial frame rate.

[0047] The initial encoding algorithm can be one of the following: AVC (Advanced Video Coding), EVC (Essential Video Coding), etc. Not all possible encoding algorithms are listed here.

[0048] The initial encoding method can be either hard encoding or soft encoding. Hard encoding refers to encoding using a non-central processing unit (CPU), such as encoding using a graphics card, chip, etc. Soft encoding refers to encoding using the CPU.

[0049] Bitrate can be defined as the number of bits per second of video data (in bps (bits per second)). Video file size = bitrate * duration. Resolution is the size of one frame of video, representing the number of pixels in width and height. Frame rate is the number of times the video refreshes per second (e.g., 25 frames per second).

[0050] The input information of the target model is the aforementioned live streaming parameters; the output information of the target model may include at least the aforementioned initial encoding parameters. The target model can be pre-trained. Specifically, the training of the target model can be performed on the server executing the data flow control method of this embodiment; or, the training of the target model can be performed on another server, in which case the server executing the data flow control method provided in this embodiment can obtain and save the aforementioned target model from the other server in advance.

[0051] As can be seen, by adopting the above scheme, the live streaming parameters sent by the first terminal can be directly processed through the target model to obtain the initial encoding parameters output by the target model. This allows for more accurate and efficient prediction of the initial encoding parameters most suitable for the first terminal.

[0052] In some possible implementations, the training of the target model can be performed on a server executing the data flow control method of this embodiment. The training method for the target model may include: training an initial model based on sample data to obtain the target model.

[0053] The sample data may include: historical live streaming parameters corresponding to historical multimedia data streams, historical encoding parameters corresponding to historical multimedia data streams, and historical quality scores corresponding to historical multimedia data streams. Furthermore, the sample data may also include: historical transcoding parameters corresponding to historical multimedia data streams.

[0054] The historical live streaming parameters corresponding to the historical multimedia data stream may include at least one of the following: historical live streaming type, historical device type. The historical live streaming type is similar to the aforementioned live streaming type and will not be repeated. Similarly, the historical device type is similar to the aforementioned device type and will not be elaborated upon.

[0055] The historical encoding parameters corresponding to the historical multimedia data stream may include at least one of the following: historical encoding algorithm, historical encoding method, historical bitrate, historical resolution, and historical frame rate. The detailed explanations of the historical encoding algorithm and historical encoding method are similar to those of the initial encoding algorithm and initial encoding method mentioned above, and will not be repeated here.

[0056] The historical transcoding parameters corresponding to the historical multimedia data stream may include at least one of the following: historical transcoding algorithm, historical transcoding resolution, and historical transcoding bitrate. For example, the historical transcoding algorithm may be one of the following: CRF (Constant Rate Factor) or CBR (Constant Bit Rate). It should be understood that there may be many other types of historical transcoding algorithms, but this embodiment does not exhaustively list them.

[0057] The historical quality score corresponding to the historical multimedia data stream may include at least one of the following: historical subjective quality score and historical auxiliary quality score.

[0058] The solution provided in this embodiment may further include: performing offline evaluation on one or more historical multimedia data streams to obtain historical subjective quality scores and historical auxiliary quality scores corresponding to each of the aforementioned one or more historical multimedia data streams; and jointly saving the historical subjective quality scores, historical auxiliary quality scores, historical live streaming parameters, historical encoding parameters, and historical transcoding parameters corresponding to each historical multimedia data stream. Correspondingly, the process of obtaining any sample data may include: when it is necessary to generate sample data using historically saved data, obtaining the historical live streaming parameters, historical encoding parameters, historical transcoding parameters, and historical quality scores corresponding to any historical multimedia data stream; and using the historical live streaming parameters, historical encoding parameters, historical transcoding parameters, and historical quality scores as sample data.

[0059] The historical subjective quality score primarily assesses the subjective picture quality of historical multimedia data streams. This historical subjective quality score can be obtained by using VMAF (Video Multi-method Assessment Fusion) to evaluate the historical multimedia data stream and obtain its historical subjective quality score. Here, VMAF is a machine learning tool, an accurate video quality assessment tool capable of predicting subjective perceptions of video quality. VMAF ensures consistency across content and maintains manageable complexity for large-scale cloud deployment. Ensuring consistency across content means that the quality assessment criteria used to predict the user's viewing experience should remain consistent when the user is watching multimedia data streams with different content (e.g., videos of different content).

[0060] Specifically, VMAF can predict image quality distortion under Hypertext Transfer Protocol (HTTP) streaming technology. Correspondingly, the aforementioned process of evaluating historical multimedia data streams using VMAF to obtain historical subjective quality scores for those streams can include: evaluating a historical multimedia data stream associated with a specific sample using VMAF to obtain an image quality distortion index for that historical multimedia data stream; and determining a historical subjective quality score for the historical multimedia data stream based on this image quality distortion index. This image quality distortion index can include compression distortion and scaling distortion indices.

[0061] The aforementioned historical auxiliary quality score can be determined by: determining the historical auxiliary quality score of the historical multimedia data stream based on its PSNR (Peak Signal Noise Rate) and / or SSIM (Structural Similarity).

[0062] The method provided in this embodiment may further include: deleting the historical multimedia data stream and its associated historical live streaming parameters, historical encoding parameters, historical transcoding parameters, and historical quality scores when the historical auxiliary quality score associated with any historical multimedia data stream is outside the preset score range; and retaining the historical multimedia data stream and its associated historical live streaming parameters, historical encoding parameters, historical transcoding parameters, and historical quality scores when the historical auxiliary quality score associated with any historical multimedia data stream is within the preset score range. In other words, if there is a difference between the historical auxiliary quality score contained in any historical multimedia data stream and the preset score range, the historical multimedia data stream can be archived as abnormal evaluation data, and then the historical multimedia data stream and its associated historical live streaming parameters, historical encoding parameters, historical transcoding parameters, and historical quality scores can be directly deleted.

[0063] The aforementioned training of the initial model based on sample data may include: selecting one or more sample data, obtaining the target sample data with the highest historical quality score from the selected one or more sample data; inputting the historical live streaming parameters in the target sample data into the initial model to obtain the prediction information output by the initial model; obtaining a loss function based on the prediction information and the historical encoding parameters and / or historical transcoding parameters in the target sample data; and updating the initial model based on the backpropagation of the loss function.

[0064] The selection of one or more sample data may include: selecting one or more sample data containing the same historical live broadcast parameters; or selecting one or more sample data containing the same historical encoding parameters.

[0065] The aforementioned method of obtaining the target sample data with the highest historical quality score from one or more sample data sets can refer to: obtaining one or more candidate sample data sets with historical quality scores higher than a specified threshold from one or more sample data sets, and sequentially using each of the one or more candidate sample data sets as the target sample data set. The specified threshold value can be configured according to actual circumstances and is not limited thereto.

[0066] The aforementioned loss function can be constructed using methods such as Minimum Mean Square Error (MSE) or Maximum Likelihood Error; the specific processing is not limited in this embodiment. The aforementioned backpropagation update of the initial model based on the loss function can be achieved by updating the model parameters of the initial model using a gradient descent algorithm based on the loss function.

[0067] The methods for determining the convergence of the aforementioned initial model may include at least one of the following: the number of training iterations reaches a preset threshold; the loss function no longer changes or is less than a specified value, etc. The preset threshold can be set according to actual conditions, such as 100 iterations, 50 iterations, etc. This embodiment does not exhaustively list all possible convergence conditions of the initial model.

[0068] Through the aforementioned processing, sample data can be constructed based on historical multimedia data streams. This sample data can include historical live streaming parameters, historical encoding parameters, historical subjective quality scores, and historical auxiliary quality scores. A loss function can be constructed based on methods such as Minimum Mean Square Error (MSE) or Maximum Likelihood Error. Then, the model parameters of the initial model are fine-tuned using the gradient descent algorithm. Finally, a target model can be obtained, which can process at least each type of live streaming parameter to obtain the optimal initial encoding parameters.

[0069] In some possible implementations, determining the target encoding parameters based on the hardware parameters and the initial encoding parameters includes: determining candidate encoding-related parameters based on the hardware parameters; and adjusting the initial encoding parameters based on the candidate encoding-related parameters to obtain the target encoding parameters if the candidate encoding-related parameters differ from the initial encoding parameters.

[0070] The hardware parameters may include at least one of the following: terminal model, acquisition component model, encoding component model, encoding component configuration method, and communication component model. It should be understood that the above is merely an illustrative description of the hardware parameters; in actual processing, there may be many more hardware parameters, but these are not exhaustively listed here.

[0071] The step of determining the candidate encoding-related parameters based on the hardware parameters may specifically include: determining the hardware score of the first terminal based on the hardware parameters; and determining the candidate encoding-related parameters corresponding to the hardware score based on a preset first correspondence.

[0072] The first correspondence may include: candidate coding-related parameters corresponding to each candidate hardware scoring range in one or more candidate hardware scoring ranges. Each candidate hardware scoring range can be set according to actual conditions. For example, it can be 0-10 points (inclusive), 10 (exclusive)-15 points (inclusive), 15 (exclusive)-20 points (inclusive). This is not exhaustive.

[0073] After determining the candidate encoding related parameters, the method may further include: determining whether the content of the candidate encoding related parameters is the same as that of the parameters of the same type in the initial encoding parameters.

[0074] Specifically, the candidate encoding-related parameters may include fewer or equal numbers of parameter types than the initial encoding parameters. For example, the candidate encoding-related parameters may include at least one of the following: candidate encoding algorithm, candidate encoding method. Correspondingly, determining whether the content of the same parameter type in the candidate encoding-related parameters is the same as that in the initial encoding parameters can be: determining whether the candidate encoding algorithm is the same as the initial encoding algorithm in the initial encoding parameters; and / or, determining whether the candidate encoding method is the same as the initial encoding method in the initial encoding parameters. It should be understood that this is merely an illustrative description of the possible contents of the candidate encoding-related parameters. In actual processing, the candidate encoding-related parameters may also include at least one of candidate bitrate, candidate resolution, etc., but this is not exhaustively listed.

[0075] For example, some types of terminals have better performance and can support the HEVC (High Efficiency Video Coding) encoding algorithm. HEVC, also known as H.265, is a high-compression, low-complexity encoding method that can maintain clarity while lowering the bitrate and saving bandwidth. Other types of terminals have poorer performance and can only use HAVC (H.264 / AVC (Advanced Video Coding)). For instance, if the first terminal is a high-performance terminal, the initial encoding algorithm in the aforementioned initial encoding parameters is HAVC, while the candidate encoding related parameters specify HEVC as the candidate encoding algorithm. If this candidate encoding algorithm differs from the initial encoding algorithm in the initial encoding parameters, then the initial encoding algorithm in the initial encoding parameters is replaced with HEVC, while the other parameters of the initial encoding parameters remain unchanged, resulting in updated initial encoding parameters. These updated initial encoding parameters are then used as the target encoding parameters.

[0076] By adopting the above scheme, the initial encoding parameters can be obtained in advance based on the live streaming parameters of the first terminal. Then, the initial encoding parameters can be further adjusted in combination with the hardware parameters of the first terminal, so as to obtain target encoding parameters that are more suitable for the first terminal and ensure the live streaming quality of the first terminal.

[0077] In some possible implementations, determining the target encoding parameter based on the hardware parameters and the initial encoding parameters includes: determining candidate encoding-related parameters based on the hardware parameters; and if the candidate encoding-related parameters are the same as the initial encoding parameters, using the initial encoding parameters as the target encoding parameters.

[0078] The specific explanation regarding determining candidate encoding-related parameters based on the aforementioned hardware parameters is the same as in the previous implementation method and will not be repeated. The process for determining whether the content of parameters of the same type as the candidate encoding-related parameters is identical to that in the initial encoding parameters is also the same as in the previous implementation method and will not be repeated.

[0079] The difference from the aforementioned implementation is that, in this implementation, when the candidate coding-related parameters are the same as the initial coding parameters, the initial coding parameters are used as the target coding parameters. For example, the candidate coding-related parameters are candidate coding algorithms A1; if the candidate coding algorithm A1 is the same as the initial coding algorithm A1 in the initial coding parameters, then the initial coding parameters are kept unchanged and directly used as the target coding parameters.

[0080] In conjunction with the foregoing embodiments, the process of determining the target encoding parameters based on the hardware parameters and the initial encoding parameters is described, which may include:

[0081] Based on the hardware parameters, determine the relevant parameters for the candidate encoding;

[0082] Determine whether the content of the candidate encoding-related parameters is the same as that of the parameters of the same type in the initial encoding parameters;

[0083] If the candidate coding-related parameters are the same as the initial coding parameters, the initial coding parameters are used as the target coding parameters; if the candidate coding-related parameters are different from the initial coding parameters, the initial coding parameters are adjusted based on the candidate coding-related parameters to obtain the target coding parameters.

[0084] In the foregoing embodiments, the target encoding parameters include parameter types that are the same as those of the initial encoding parameters. For example, if the initial encoding parameters may include at least one of the following: initial encoding algorithm, initial encoding method, initial bitrate, and initial resolution, then the target encoding parameters may also include at least one of the following: target encoding algorithm, target encoding method, target bitrate, and target resolution.

[0085] Thus, based on the live streaming parameters and the target model, and considering the hardware parameters of the first terminal used in this live stream, if it is determined that the initial encoding parameters do not need to be adjusted, the initial encoding parameters can be directly used as the target encoding parameters. Through secondary verification, it can be ensured that the target encoding parameters are the most suitable parameters for the first terminal, thus guaranteeing the live streaming quality of the first terminal.

[0086] In some possible implementations, the method may further include: acquiring the current network parameters of the first terminal in real time; determining the network change status of the first terminal based on the current network parameters and the historical network parameters of the first terminal; determining the adjusted target encoding parameters if the target encoding parameters need to be adjusted based on the network change status of the first terminal; and sending the adjusted target encoding parameters to the first terminal.

[0087] Additionally, it may include: if it is determined based on the network change status of the first terminal that the target encoding parameters do not need to be adjusted, keeping the target encoding parameters unchanged; and continuing to acquire the current network parameters of the first terminal in real time.

[0088] Furthermore, after sending the adjusted target encoding parameters to the first terminal, the original target encoding parameters can be deleted, and the adjusted target encoding parameters can be used as the current target encoding parameters; then the current network parameters of the first terminal can continue to be acquired in real time.

[0089] The current network parameters can refer to the current network speed. Similarly, the historical network parameters can refer to historical network speeds. These historical network speeds can be the network speeds at a specified time interval prior to the current moment. The specified time interval can be set according to actual conditions, such as 10 minutes, 15 minutes, 1 minute, or longer or shorter; an exhaustive list is not provided here.

[0090] It should be noted that the current network parameters and historical network parameters may also include other types of network parameters. For example, the current network parameter may include the current network bandwidth, etc. This document does not exhaustively list all possible types of network parameters; as long as the current network parameter and historical network parameter are of the same type, they are within the scope of protection of this embodiment. The current network bandwidth may be the uplink bandwidth of the current network.

[0091] Taking the aforementioned current network parameters as the current network speed and the aforementioned historical network parameters as the historical network speed as an example, the following explanation is provided. Determining the network change status of the first terminal based on its current network parameters and historical network parameters may include: determining the network speed difference between the current network speed and the historical network speed of the first terminal, and determining the network change status of the first terminal based on the network speed difference.

[0092] The step of determining the network change status of the first terminal based on the network speed difference may include: determining the network change status of the first terminal as improved when the network speed difference is higher than a first threshold; determining the network change status of the first terminal as worse when the network speed difference is lower than a second threshold; and determining the network change status of the first terminal as unchanged when the network speed difference is neither higher than the first threshold nor lower than the second threshold. The first threshold is higher than the second threshold. The second threshold can be a negative number.

[0093] The above example illustrates the concept of current network speed as the current network parameter and historical network speed as the historical network parameter. In actual processing, the current network parameter may include the current network speed and / or current bandwidth, and the historical network parameter may include the historical network speed and / or historical bandwidth. The specific comparison and processing methods are similar to the aforementioned example, and therefore will not be exhaustive. The historical bandwidth may refer to the historical network uplink bandwidth.

[0094] After determining the network change status of the first terminal, it can be further determined whether the target encoding parameters need to be adjusted based on the network change status of the first terminal. Specifically, this can include: if the network change status of the first terminal is that the network is improving and the duration reaches a preset duration, it can be determined that the target encoding parameters need to be adjusted; if the network change status of the first terminal is that the network is deteriorating and the duration reaches a preset duration, it can be determined that the target encoding parameters need to be adjusted. In addition, in other cases besides the above two situations, it can be determined that no adjustment of the target encoding parameters is required.

[0095] The preset duration can be set according to the actual situation, such as 5 minutes, 10 minutes, or longer or shorter.

[0096] The determination of the adjusted target coding parameters may include: determining the adjusted target coding parameters based on the network change status of the first terminal and the target coding parameters.

[0097] Specifically, determining the adjusted target coding parameters based on the network change status of the first terminal and the target coding parameters may include at least one of the following:

[0098] If the network status of the first terminal is deterioration, then the target bitrate in the target coding parameters is reduced by one bitrate level to obtain the adjusted target bitrate. The adjusted target bitrate is then used to replace the target bitrate to obtain the adjusted target coding parameters.

[0099] If the network status of the first terminal is deterioration, then the target resolution in the target encoding parameters is reduced by one resolution level to obtain the adjusted target resolution. The target resolution is then replaced with the adjusted target resolution to obtain the adjusted target encoding parameters.

[0100] If the network status of the first terminal changes to "network improved", then the target bitrate in the target encoding parameters is increased by one bitrate level to obtain the adjusted target bitrate. The adjusted target bitrate is then used to replace the target bitrate to obtain the adjusted target encoding parameters.

[0101] If the network status of the first terminal changes to "network improved", then the target resolution in the target encoding parameters is increased by one resolution level to obtain the adjusted target resolution. The adjusted target resolution is then used to replace the target resolution to obtain the adjusted target encoding parameters.

[0102] It should be understood that after reducing the target bitrate in the target coding parameters by one bitrate level to obtain the adjusted target bitrate, the process may further include: determining whether the adjusted target bitrate is lower than the minimum bitrate threshold. If it is lower, the target coding parameters are not adjusted; otherwise, the adjusted target bitrate can be used to replace the target bitrate to obtain the adjusted target coding parameters.

[0103] After reducing the target resolution in the target encoding parameters by one resolution level to obtain the adjusted target resolution, the method may further include: determining whether the adjusted target resolution is lower than the minimum resolution threshold. If it is lower, the target encoding parameters are not adjusted; otherwise, the target resolution can be replaced based on the adjusted target resolution to obtain the adjusted target encoding parameters.

[0104] After increasing the target bitrate in the target encoding parameters by one bitrate level to obtain the adjusted target bitrate, the method may further include: determining whether the adjusted target bitrate is higher than the highest bitrate threshold. If it is higher, the target encoding parameters are not adjusted; otherwise, the target bitrate can be replaced based on the adjusted target bitrate to obtain the adjusted target encoding parameters.

[0105] After increasing the target resolution in the target encoding parameters by one resolution level to obtain the adjusted target resolution, the method may further include: determining whether the adjusted target resolution is higher than the highest resolution threshold. If it is higher, the target encoding parameters are not adjusted; otherwise, the target resolution can be replaced based on the adjusted target resolution to obtain the adjusted target encoding parameters.

[0106] The aforementioned minimum bit rate threshold is lower than the maximum bit rate threshold, and the aforementioned minimum resolution threshold is lower than the maximum resolution threshold.

[0107] The aforementioned resolution and bitrate levels can be preset according to actual conditions and are not limited. For example, if it is determined that the network speed of the first terminal is continuously decreasing, indicating a deterioration in network status, the resolution level can be lowered by one level. For instance, if the target resolution is 1080p, lowering it by one level will result in a target resolution of 720p; if the target resolution is 720p, lowering it by one level will result in a target resolution of 540p.

[0108] As can be seen, by adopting the above scheme, after the target encoding parameters are determined, the target encoding parameters can be adjusted in real time based on the current network changes of the first terminal, so that the first terminal adopts the adjusted target encoding parameters that are more compatible with the current network conditions, and ultimately ensure the live broadcast quality of the first terminal.

[0109] In some possible implementations, it may also include: determining the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

[0110] Specifically, determining the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters may include: inputting the live streaming parameters into the target model to obtain the initial transcoding parameters output by the target model; determining candidate transcoding-related parameters based on the hardware parameters; determining the target transcoding parameters based on the candidate transcoding-related parameters and the initial transcoding parameters; and using the target transcoding parameters as the current transcoding parameters corresponding to the first terminal.

[0111] The step of determining the target transcoding parameter based on the candidate transcoding-related parameters and the initial transcoding parameter may include: determining whether the candidate transcoding-related parameters are the same as the initial transcoding parameter; if the candidate transcoding-related parameters are the same as the initial transcoding parameter, using the initial transcoding parameter as the target transcoding parameter; if the candidate transcoding-related parameters are different from the initial transcoding parameter, adjusting the initial transcoding parameter based on the candidate transcoding-related parameters to obtain the target transcoding parameter.

[0112] In other words, given the input live streaming parameters, the output information of the aforementioned target model may include the aforementioned initial encoding parameters, and may also include initial transcoding parameters. The initial transcoding parameters may include at least one of the following: initial transcoding algorithm, initial transcoding bitrate, and initial transcoding resolution.

[0113] The target model has been described in detail in the foregoing embodiments, and the hardware parameters have also been described in detail in the foregoing embodiments, so they will not be repeated here.

[0114] The step of determining candidate transcoding-related parameters based on the hardware parameters may specifically include: determining the hardware score of the first terminal based on the hardware parameters; and determining the candidate transcoding-related parameters corresponding to the hardware score based on a preset second correspondence.

[0115] The second correspondence may include: candidate transcoding-related parameters corresponding to each candidate hardware scoring range within one or more candidate hardware scoring ranges. The description of each candidate hardware scoring range is the same as in the foregoing implementation, and will not be repeated here.

[0116] The candidate transcoding-related parameters may contain fewer or fewer parameter types than the initial transcoding parameters. For example, the candidate transcoding-related parameters may include candidate transcoding algorithms. For instance, if the candidate transcoding-related parameters are candidate transcoding algorithm C1 and the initial transcoding algorithm is C2, and the candidate transcoding algorithm C1 differs from the initial transcoding algorithm C2 in the initial transcoding parameters, then the initial transcoding algorithm in the initial transcoding parameters is replaced with C1 to obtain the target transcoding parameters.

[0117] The target transcoding parameters may include the same parameter types as the initial transcoding parameters. For example, the target transcoding parameters may include at least one of: target transcoding bitrate, target transcoding resolution, and target transcoding algorithm. It should be understood that the aforementioned target transcoding bitrate may be a maximum transcoding bitrate or an optimal transcoding bitrate level; or, the aforementioned target transcoding bitrate may be one or more transcoding sub-bitrates, where different transcoding sub-bitrates are used to obtain multimedia data streams with different bitrates. The aforementioned target transcoding resolution may also be a maximum transcoding resolution or an optimal transcoding resolution; or, the aforementioned target transcoding resolution may be one or more transcoding sub-resolutions, where different transcoding sub-resolutions are used to obtain multimedia data streams with different resolutions.

[0118] It should also be noted that the method provided in this embodiment may further include: when the adjusted target encoding parameters are determined, adjusting the target transcoding parameters based on the adjusted target encoding parameters to obtain the adjusted target transcoding parameters; and using the adjusted target transcoding parameters as the current transcoding parameters corresponding to the first terminal.

[0119] Specifically, adjusting the target transcoding parameters based on the adjusted target encoding parameters to obtain the adjusted target transcoding parameters can be as follows: If the target bitrate changes in the adjusted target encoding parameters, adjust the target transcoding bitrate in the target transcoding parameters based on the target bitrate, and use the target transcoding parameters containing the adjusted target transcoding bitrate as the adjusted target transcoding parameters. And / or, if the target resolution changes in the adjusted target encoding parameters, adjust the target transcoding resolution in the target transcoding parameters based on the target resolution, and use the target transcoding parameters containing the adjusted target transcoding resolution as the adjusted target transcoding parameters.

[0120] The aforementioned adjustment of the target transcoding bitrate in the target transcoding parameters based on the target bitrate can refer to directly using the target bitrate as the target transcoding bitrate. And / or, adjusting the target transcoding resolution in the target transcoding parameters based on the target resolution can refer to directly using the target resolution as the target transcoding resolution.

[0121] By adopting the above scheme, the initial transcoding parameters can be obtained in advance based on the live streaming parameters of the first terminal. Then, the initial transcoding parameters can be further adjusted in combination with the hardware parameters of the first terminal, so as to obtain target transcoding parameters that are more suitable for the first terminal. This ensures the transcoding quality of the encoded multimedia data stream uploaded by the first terminal, and ultimately ensures the quality of the multimedia data stream played on the playback end.

[0122] In some possible implementations, the method may further include: sending the current transcoding parameters corresponding to the first terminal to a transcoding server; wherein the current transcoding parameters corresponding to the first terminal are used by the transcoding server to transcode the encoded multimedia data stream uploaded by the first terminal to obtain a multimedia data stream of each of N bitrates; N is an integer greater than or equal to 1. In a preferred example, N may be greater than or equal to 2.

[0123] That is, after obtaining the current transcoding parameters corresponding to the first terminal based on the aforementioned implementation method, the current transcoding parameters corresponding to the first terminal can be directly sent to the transcoding server so that the transcoding server can process the encoded multimedia data stream of the first terminal.

[0124] By adopting the above scheme, the current transcoding parameters that are more suitable for the first terminal can be sent to the transcoding server to ensure the transcoding quality of the encoded multimedia data stream uploaded by the first terminal, and ultimately ensure the quality of the multimedia data stream played on the playback end.

[0125] In some other possible implementations, the method may further include: upon receiving the encoded multimedia data stream uploaded by the first terminal, transcoding the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain multimedia data streams at each of N bitrates; where N is an integer greater than or equal to 1; and sending the multimedia data streams at each of the N bitrates to an edge server. In a preferred example, N may be greater than or equal to 2.

[0126] Unlike the aforementioned implementation method, the server executing the data flow control method of this embodiment also has a transcoding function. Therefore, after obtaining the current transcoding parameters corresponding to the first terminal, it is not necessary to send out the current transcoding parameters corresponding to the first terminal. It is only necessary to transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal when receiving the encoded multimedia data stream uploaded by the first terminal, so as to obtain multimedia data streams of each bitrate among N bitrates.

[0127] For example, the encoded multimedia data stream sent by the first terminal is a 1080 video stream. The current transcoding parameters corresponding to the first terminal include an optimal transcoding bitrate level. Assuming that the optimal transcoding bitrate level is 1080, the received 1080 video stream can be transcoded to obtain three multimedia data streams with different bitrates: ultra-high definition (1080), high definition (720), and standard definition (540), thereby meeting the viewing needs of different playback terminals.

[0128] The aforementioned edge server can refer to the edge nodes in a CDN (Content Delivery Network). Accordingly, the playback client can obtain a multimedia data stream with a bitrate that matches its own network conditions from the edge node and play it, based on the actual network situation.

[0129] The multimedia data stream described in this embodiment may include: video stream and / or audio stream.

[0130] By adopting the above scheme, the current transcoding parameters more suitable for the first terminal can be used to transcode the encoded multimedia data stream uploaded by the first terminal, and obtain multimedia data streams with different bitrates at one or more bitrates. This ensures that the multimedia data stream with the matching bitrate is selected for playback at the playback end, thereby guaranteeing the playback quality at the playback end.

[0131] A second aspect of this disclosure provides a data flow control system, such as... Figure 2 As shown, it includes:

[0132] The data stream control device 201 is used to receive parameter information sent by a first terminal; wherein the parameter information includes live streaming parameters and hardware parameters; determine initial encoding parameters based on the live streaming parameters; determine target encoding parameters based on the hardware parameters and the initial encoding parameters; and send the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain an encoded multimedia data stream.

[0133] The first terminal 202 is used to send the parameter information and receive the target encoded parameters.

[0134] In some possible implementations, the data flow control device is used to input the live streaming parameters into the target model to obtain the initial encoding parameters output by the target model.

[0135] In some possible implementations, the training of the target model can be performed by the data flow control device. The training method for the target model is the same as that described in the first aspect embodiment above, and will not be repeated here.

[0136] In some possible implementations, the data flow control device is used to train an initial model based on sample data to obtain the target model; the sample data includes: historical live streaming parameters corresponding to historical multimedia data streams, historical encoding parameters corresponding to historical multimedia data streams, and historical quality scores corresponding to historical multimedia data streams.

[0137] In some possible implementations, the data flow control device is configured to determine candidate coding-related parameters based on the hardware parameters; and, if the candidate coding-related parameters differ from the initial coding parameters, adjust the initial coding parameters based on the candidate coding-related parameters to obtain the target coding parameters.

[0138] In some possible implementations, the data flow control device is used to determine candidate coding-related parameters based on the hardware parameters; if the candidate coding-related parameters are the same as the initial coding parameters, the initial coding parameters are used as the target coding parameters.

[0139] In some possible implementations, the data flow control device is configured to acquire the current network parameters of the first terminal in real time; determine the network change status of the first terminal based on the current network parameters and the historical network parameters of the first terminal; determine the adjusted target encoding parameters if the target encoding parameters need to be adjusted based on the network change status of the first terminal; send the adjusted target encoding parameters to the first terminal; and the first terminal is configured to receive the adjusted target encoding parameters.

[0140] In some possible implementations, the data flow control device is used to determine the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

[0141] In some possible implementations, such as Figure 3 As shown, the system also includes:

[0142] The transcoding server 301 is used to receive the current transcoding parameters corresponding to the first terminal; the current transcoding parameters are used to transcode the encoded multimedia data stream uploaded by the first terminal to obtain a multimedia data stream corresponding to each of N bitrates, where N is an integer greater than or equal to 1.

[0143] The data flow control device 201 is used to send the current transcoding parameters corresponding to the first terminal.

[0144] The system also includes: an edge server 302, used to receive multimedia data streams corresponding to each of the N bitrates;

[0145] The second terminal 303 is used to obtain the multimedia data stream with a matching bit rate from the multimedia data stream corresponding to each of the N bit rates.

[0146] The first terminal 202 is configured to, upon acquiring the original multimedia data stream, encode the original multimedia data stream based on current encoding parameters to obtain an encoded multimedia data stream; and send the encoded multimedia data stream to the transcoding server; wherein the current encoding parameters are the target encoding parameters or the adjusted target encoding parameters;

[0147] The transcoding server 301 is configured to, upon receiving the encoded multimedia data stream, transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain a multimedia data stream corresponding to each of the N bitrates; and send the multimedia data stream corresponding to each of the N bitrates.

[0148] It should be understood that the aforementioned first terminal may upload the encoded multimedia data stream to the transcoding server 301 via the edge server 302. Alternatively, the aforementioned first terminal may directly upload the encoded multimedia data stream to the transcoding server 301.

[0149] Combination Figure 4 An exemplary illustration of the architecture of the aforementioned data flow control system is provided below:

[0150] The aforementioned data flow control device can be a device installed in a server. For example, the server equipped with the data flow control device can be... Figure 4 The central control server 410 is shown in the diagram. The target model can be configured on the central control server 410. The functions of this target model have been described in detail in the preceding embodiments and will not be repeated here.

[0151] The first terminal 420 may be a terminal equipped with a camera, virtual camera, microphone, etc. The current encoding parameter of the first terminal may be a target encoding parameter or an adjusted target encoding parameter. For example, if the first terminal only receives the target encoding parameter, it can use the target encoding parameter as the current encoding parameter. If the first terminal receives an adjusted target encoding parameter, it can use the adjusted target encoding parameter as the current encoding parameter. It should be understood that the first terminal can be configured to save only one set or group of current encoding parameters at any given time. For example, after receiving an adjusted target encoding parameter, the previously saved target encoding parameter is deleted, and the adjusted target encoding parameter is used as the current encoding parameter. Furthermore, when the first terminal receives the next adjusted target encoding parameter, the previously saved adjusted target encoding parameter is deleted, and the currently received adjusted target encoding parameter is saved as the current encoding parameter.

[0152] The first terminal 420 can be set Figure 4 The encoding module shown can be used to encode the raw multimedia data stream collected by the first terminal in real time and transmit the encoded multimedia data stream to the live streaming source station. Here, the raw multimedia data stream includes the raw video stream and / or the raw audio stream; wherein, the raw multimedia data stream can be collected by a camera installed on the first terminal, or by forwarding a network video source. The encoding module in the first terminal can communicate with the central control server 410 in real time, transmitting live streaming parameters (live streaming event, live streaming data, live streaming type, live streaming mode) and hardware parameters to the central control server 410, receiving target encoding parameters from the central control server 410, and can also dynamically adjust the current encoding parameters in real time according to the adjusted target encoding parameters issued by the central control server 410.

[0153] It should be noted that the encoding module of the first terminal 420 can transmit the encoded multimedia data stream to the live streaming source station through the edge server in CDN430.

[0154] like Figure 4As shown, there can be one or more playback terminals 440, and the aforementioned second terminal can be any one of the one or more playback terminals 440. Since the processing for each playback terminal is the same as that for the second terminal, only the bitrate determined by different playback terminals based on their current network status may be different, which will not be elaborated here.

[0155] The second terminal is used to obtain its own matching bitrate based on the current network status and retrieve the multimedia data stream with its matching bitrate from the edge server in the CDN430. Specifically, this second terminal is used to determine the current network status based on actual stuttering events; if a stuttering event occurs, the resolution level is automatically downgraded by one level, and the multimedia data stream with its own matching bitrate is determined based on the downgraded level; if no stuttering event occurs, the resolution level can be increased by one level, and the multimedia data stream with its own matching bitrate is determined based on the increased level.

[0156] The edge server can be a node in a CDN430. A CDN430 can include one or more edge servers, and any one of the edge servers can also be called an edge node.

[0157] The aforementioned transcoding server can also be called Figure 4 The live streaming source server 450 shown here may also be called by other names, which will not be listed here; unless otherwise specified below, "live streaming source server" and "transcoding server" have the same meaning. Taking the transcoding server specifically as live streaming source server 450 as an example, live streaming source server 450 can be configured with... Figure 4 The transcoding module shown allows the live streaming origin server 450 to receive encoded multimedia data streams and transcode them in real time, ultimately obtaining multi-bitrate multimedia data streams, which are then sent to the edge server in CDN 430, enabling the edge server to respond to the playback requests of one or more playback terminals.

[0158] The live streaming source station 450 communicates in real time with the central control server 410 through the transcoding module to receive the current transcoding parameters corresponding to the first terminal sent by the central control server 410. Then, the transcoding module of the live streaming source station 450 transcodes the encoded multimedia data stream uploaded by the first terminal based on the current transcoding parameters.

[0159] Both the aforementioned first terminal 420 and live streaming source station 450 can establish connections with the central control server 410 via WebSocket (network socket) long-lived connections. In this way, the central control server 410 can promptly send information such as current transcoding parameters and target encoding parameters to the first terminal 420 and live streaming source station 450 via messages. Furthermore, because WebSocket can maintain a long-lived connection heartbeat mechanism, the connectivity between the first terminal 420, the live streaming source station 450, and the central control server 410 can be guaranteed.

[0160] by Figure 4 Taking the architecture shown as an example, combined with Figure 5 An exemplary illustration of a data flow control method is provided below:

[0161] S5011, the first terminal 420 sends parameter information to the central control server 410; the parameter information may include live broadcast parameters and hardware parameters.

[0162] S5012, the central control server 410 determines the initial encoding parameters based on the live streaming parameters, determines the target encoding parameters based on the hardware parameters and the initial encoding parameters, and determines the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

[0163] S5013, the central control server 410 sends the target encoding parameters to the first terminal 410. The target encoding parameters are used to encode the original multimedia data stream to obtain an encoded multimedia data stream.

[0164] S5014, the central control server 410 sends the current transcoding parameters corresponding to the first terminal to the live streaming source station 450.

[0165] The processing of S5013 and S5014 above can be performed in any order.

[0166] For example, when a broadcaster creates a live streaming room on the first terminal, they can upload live streaming parameters (such as the start date, live streaming type, device type, etc.) to the central control server 410. The central control server 410 inputs the live streaming parameters into the target model to obtain initial encoding parameters, which can also be called optimal encoding prediction parameters. At the same time, the central control server 410 can also obtain the initial transcoding parameters output by the target model.

[0167] Then, the central control server 410 uses the hardware parameters reported by the first terminal to obtain a hardware score for the first terminal, and then provides suitable candidate encoding parameters for the first terminal. For example, some terminals have better performance and support HEVC, while others support HAVC; therefore, the initial encoding parameters can be adjusted based on the terminal's hardware parameters to finally determine the target encoding parameters.

[0168] After the above processing is completed, the first terminal can obtain the target encoding parameters; at the same time, the live streaming source station 450 can obtain the current transcoding parameters.

[0169] S5021, when the first terminal 410 acquires the original multimedia data stream, it encodes the original multimedia data stream based on the current encoding parameters to obtain the encoded multimedia data stream.

[0170] For example, the broadcaster needs to use a first terminal 410 with a camera or a virtual camera to collect source audio and video, encode the source audio and video, and then push the encoded audio and video to the live streaming source station 450; where the source audio and video is the aforementioned original multimedia data stream; and the encoded audio and video is the aforementioned encoded multimedia data stream.

[0171] S5022, the first terminal 410 sends the encoded multimedia data stream to the live streaming source station 450. Here, although Figure 5 Although not explicitly shown, the first terminal 410 could send the encoded multimedia data stream to the live streaming origin 450 via an edge server in the CDN. Alternatively, it could be as follows: Figure 5 As shown, the first terminal 410 directly sends the encoded multimedia data stream to the live streaming source station 450.

[0172] S5023, when the live streaming source station 450 receives the encoded multimedia data stream, it transcodes the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain the multimedia data stream corresponding to each of the N bitrates.

[0173] S5024, the live streaming origin 450 sends the multimedia data stream corresponding to each of the N bitrates to the edge server in CDN 430.

[0174] S5025, the second terminal obtains a multimedia data stream with a matching bitrate from the edge server in CDN430.

[0175] During the execution of S5021 to S5025, the following processes may also be performed:

[0176] S5031, the central control server 410 obtains the current network parameters of the first terminal in real time;

[0177] S5032, the central control server 410 determines the network change status of the first terminal based on the current network parameters of the first terminal and the historical network parameters of the first terminal;

[0178] S5033, when the central control server 410 determines that the target encoding parameters need to be adjusted based on the network change status of the first terminal, the adjusted target encoding parameters are determined.

[0179] S5034, the central control server 410 sends the adjusted target encoding parameters to the first terminal 420;

[0180] S5035, the first terminal 420 receives the adjusted target encoding parameters, uses the adjusted target encoding parameters as the current encoding parameters, and then continues to execute the aforementioned S5021.

[0181] This is because the first terminal may be in an outdoor or mobile environment, so the network status may be unstable. In this case, if you want to ensure smooth playback, you need to dynamically optimize the target encoding parameters (such as the target encoding bitrate and / or target resolution) in real time according to the network status of the first terminal. The central control server 410 obtains the current network parameters of the first terminal in real time to determine the network change status and adjusts the target encoding parameters according to the network change status.

[0182] In some other possible implementations, Figure 2 On the basis of, such as Figure 6 As shown, the system also includes:

[0183] Edge server 601 is used to receive multimedia data streams corresponding to each of N bitrates;

[0184] The second terminal 602 is used to obtain the multimedia data stream with a matching bit rate from the multimedia data stream corresponding to each of the N bit rates.

[0185] The first terminal 202 is configured to, upon acquiring the original multimedia data stream, encode the original multimedia data stream based on current encoding parameters to obtain an encoded multimedia data stream; and send the encoded multimedia data stream to the data stream control device; wherein the current encoding parameters are the target encoding parameters or the adjusted target encoding parameters;

[0186] The data stream control device 201 is used to, upon receiving the encoded multimedia data stream, transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain a multimedia data stream corresponding to each of the N bitrates; and send the multimedia data stream corresponding to each of the N bitrates.

[0187] It should be understood that the aforementioned first terminal may upload the encoded multimedia data stream to the data flow control device 201 through an edge server; or it may directly upload the encoded multimedia data stream to the data flow control device 201.

[0188] In this embodiment Figure 6 Based on the system architecture shown, combined with Figure 7 An exemplary illustration of a data flow control method is provided below:

[0189] S7011, the first terminal sends parameter information to the data flow control device; the parameter information may include live streaming parameters and hardware parameters.

[0190] S7012, the data flow control device determines the initial encoding parameters based on the live streaming parameters, determines the target encoding parameters based on the hardware parameters and the initial encoding parameters, and determines the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

[0191] S7013, the data flow control device sends the target encoding parameters to the first terminal. The target encoding parameters are used to encode the original multimedia data stream to obtain an encoded multimedia data stream.

[0192] S7021, when the first terminal acquires the original multimedia data stream, it encodes the original multimedia data stream based on the current encoding parameters to obtain the encoded multimedia data stream.

[0193] S7022, the first terminal sends the encoded multimedia data stream to the data stream control device.

[0194] S7023, when the data flow control device receives the encoded multimedia data stream, it transcodes the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain the multimedia data stream corresponding to each of the N bitrates.

[0195] S7024, the data flow control device sends the multimedia data stream corresponding to each of the N bitrates to the edge server.

[0196] S7025, the second terminal obtains a multimedia data stream with a matching bitrate from the edge server.

[0197] Figure 7 In the scenario shown, the method by which the data flow control device obtains the adjusted target encoding parameters and the processing by which the first terminal uses the adjusted target encoding parameters as the current encoding parameters may include:

[0198] S7031, the data flow control device obtains the current network parameters of the first terminal in real time;

[0199] S7032, the data flow control device determines the network change status of the first terminal based on the current network parameters of the first terminal and the historical network parameters of the first terminal;

[0200] S7033, when the data flow control device determines that the target coding parameters need to be adjusted based on the network change status of the first terminal, the adjusted target coding parameters are determined.

[0201] S7034, the data flow control device sends the adjusted target encoding parameters to the first terminal;

[0202] S7035, the first terminal receives the adjusted target encoding parameters, uses the adjusted target encoding parameters as the current encoding parameters, and then continues to execute the aforementioned S7021.

[0203] The beneficial effects of the solution provided in this embodiment are explained in conjunction with related technologies: In related technologies, a range-based bitrate setting is typically used to uniformly optimize encoding and / or transcoding parameters. For example, a bitrate of 2.5M is used for 720p resolution, and 5M for 1080p. However, in actual live streaming scenarios, each terminal is different. For example, each terminal has different network environments, live streaming parameters, and hardware parameters. Therefore, using a uniform adjustment method may result in wasted bandwidth costs and may also fail to guarantee quality. The solution provided in this embodiment can combine the live streaming parameters and hardware parameters of the first terminal to determine the target encoding parameters corresponding to that first terminal. In this way, encoding parameters that are more in line with the requirements of different terminals can be determined, so that each terminal can encode multimedia data streams based on encoding parameters that meet its own requirements when conducting live streaming, ensuring the quality of multimedia data streams uploaded by the terminals while using reasonable bandwidth.

[0204] A third aspect of this disclosure provides a data flow control device, such as... Figure 8 As shown, it includes:

[0205] The receiving module 801 is used to receive parameter information sent by the first terminal; wherein the parameter information includes live streaming parameters and hardware parameters;

[0206] Processing module 802 is used to determine initial encoding parameters based on the live streaming parameters; and to determine target encoding parameters based on the hardware parameters and the initial encoding parameters.

[0207] The sending module 803 is used to send the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream.

[0208] The processing module is used to input the live streaming parameters into the target model to obtain the initial encoding parameters output by the target model.

[0209] exist Figure 8 On the basis of, such as Figure 9 As shown, the device further includes:

[0210] Training module 901 is used to train the initial model based on sample data to obtain the target model; the sample data includes: historical live streaming parameters corresponding to the historical multimedia data stream, historical encoding parameters corresponding to the historical multimedia data stream, and historical quality scores corresponding to the historical multimedia data stream.

[0211] The processing module is configured to determine candidate encoding-related parameters based on the hardware parameters; and, if the candidate encoding-related parameters differ from the initial encoding parameters, adjust the initial encoding parameters based on the candidate encoding-related parameters to obtain the target encoding parameters.

[0212] The processing module is used to determine candidate encoding-related parameters based on the hardware parameters; if the candidate encoding-related parameters are the same as the initial encoding parameters, the initial encoding parameters are used as the target encoding parameters.

[0213] The device further includes: an adjustment module 902, configured to determine the network change status of the first terminal based on the current network parameters of the first terminal and the historical network parameters of the first terminal; and to determine the adjusted target encoding parameters if it is determined that the target encoding parameters need to be adjusted based on the network change status of the first terminal.

[0214] The receiving module 801 is used to acquire the current network parameters of the first terminal in real time;

[0215] The sending module 803 is used to send the adjusted target encoding parameters to the first terminal.

[0216] The processing module 802 is used to determine the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

[0217] The sending module is used to send the current transcoding parameters corresponding to the first terminal to the transcoding server; wherein, the current transcoding parameters are used by the transcoding server to transcode the encoded multimedia data stream uploaded by the first terminal to obtain a multimedia data stream of each of N bitrates; N is an integer greater than or equal to 1.

[0218] The device further includes: a transcoding module 903, configured to, upon receiving the encoded multimedia data stream uploaded by the first terminal, transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain a multimedia data stream of each of N bitrates; N is an integer greater than or equal to 1;

[0219] The sending module 803 is used to send the multimedia data stream of each of the N bitrates to the edge server.

[0220] The data flow control device provided in this embodiment can be installed in a server. The specific processing of each module in the device of this embodiment is the same as that in the aforementioned data flow control method, and will not be repeated here.

[0221] The acquisition, storage, and application of user personal information involved in the technical solution disclosed herein comply with the provisions of relevant laws and regulations and do not violate public order and good morals.

[0222] According to embodiments of this disclosure, this disclosure also provides an electronic device, a readable storage medium, and a computer program product.

[0223] Figure 10 A schematic block diagram of an example electronic device 1000 that can be used to implement embodiments of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device may also represent various forms of mobile devices, such as personal digital processors, cellular phones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely illustrative and are not intended to limit the implementation of the present disclosure described and / or claimed herein.

[0224] like Figure 10 As shown, the electronic device 1000 includes a computing unit 1001, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 1002 or a computer program loaded from a storage unit 1008 into a random access memory (RAM) 1003. The RAM 1003 may also store various programs and data required for the operation of the electronic device 1000. The computing unit 1001, ROM 1002, and RAM 1003 are interconnected via a bus 1004. An input / output (I / O) interface 1005 is also connected to the bus 1004.

[0225] Multiple components in electronic device 1000 are connected to I / O interface 1005, including: input unit 1006, such as keyboard, mouse, etc.; output unit 1007, such as various types of displays, speakers, etc.; storage unit 1008, such as disk, optical disk, etc.; and communication unit 1009, such as network card, modem, wireless transceiver, etc. Communication unit 1009 allows electronic device 1000 to exchange information / data with other devices through computer networks such as the Internet and / or various telecommunications networks.

[0226] The computing unit 1001 can be a variety of general-purpose and / or special-purpose processing components with processing and computing capabilities. Some examples of the computing unit 1001 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various special-purpose artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, a digital signal processor (DSP), and any suitable processor, controller, microcontroller, etc. The computing unit 1001 performs the various methods and processes described above. For example, in some embodiments, the various methods described above can be implemented as computer software programs tangibly contained in a machine-readable medium, such as storage unit 1008. In some embodiments, part or all of the computer program can be loaded and / or installed on the electronic device 1000 via ROM 1002 and / or communication unit 1009. When the computer program is loaded into RAM 1003 and executed by the computing unit 1001, one or more steps of the various methods described above can be performed. Alternatively, in other embodiments, the computing unit 1001 can be configured to perform the various methods described above by any other suitable means (e.g., by means of firmware).

[0227] Various embodiments of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field-programmable gate arrays (FPGAs), application-specific integrated circuits (ASICs), application-specific standard products (ASSPs), systems-on-a-chip (SoCs), payload-programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various embodiments may include implementations in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which may be a dedicated or general-purpose programmable processor, capable of receiving data and instructions from a storage system, at least one input device, and at least one output device, and transmitting data and instructions to the storage system, the at least one input device, and the at least one output device.

[0228] The program code used to implement the methods of this disclosure may be written in any combination of one or more programming languages. This program code may be provided to a processor or controller of a general-purpose computer, special-purpose computer, or other programmable data processing apparatus, such that when executed by the processor or controller, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a machine, partially on a machine, as a standalone software package partially on a machine and partially on a remote machine, or entirely on a remote machine or server.

[0229] In the context of this disclosure, a machine-readable medium can be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, apparatus, or device. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can be, but is not limited to, electronic, magnetic, optical, electromagnetic, infrared, or semiconductor systems, apparatus, or devices, or any suitable combination of the foregoing. More specific examples of machine-readable storage media include electrical connections based on one or more wires, portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), optical fiber, portable compact disk read-only memory (CD-ROM), optical storage devices, magnetic storage devices, or any suitable combination of the foregoing.

[0230] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device for displaying information to the user (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor); and a keyboard and pointing device (e.g., a mouse or trackball) through which the user provides input to the computer. Other types of devices can also be used to provide interaction with the user; for example, feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including sound input, voice input, or tactile input).

[0231] The systems and technologies described herein can be implemented in computing systems that include backend components (e.g., as a data server), or computing systems that include middleware components (e.g., an application server), or computing systems that include frontend components (e.g., a user computer with a graphical user interface or web browser through which a user can interact with embodiments of the systems and technologies described herein), or any combination of such backend, middleware, or frontend components. The components of the system can be interconnected via digital data communication of any form or medium (e.g., a communication network). Examples of communication networks include local area networks (LANs), wide area networks (WANs), and the Internet.

[0232] Computer systems can include clients and servers. Clients and servers are generally located far apart and typically interact via communication networks. Client-server relationships are created by computer programs running on the respective computers and having a client-server relationship with each other. Servers can be cloud servers, servers in distributed systems, or servers incorporating blockchain technology.

[0233] It should be understood that the various forms of processes shown above can be used to rearrange, add, or delete steps. For example, the steps described in this disclosure can be executed in parallel, sequentially, or in different orders, as long as the desired result of the technical solution disclosed in this disclosure can be achieved, and this is not limited herein.

[0234] The specific embodiments described above do not constitute a limitation on the scope of protection of this disclosure. Those skilled in the art should understand that various modifications, combinations, sub-combinations, and substitutions can be made according to design requirements and other factors. Any modifications, equivalent substitutions, and improvements made within the spirit and principles of this disclosure should be included within the scope of protection of this disclosure.

Claims

1. A data flow control method, comprising: Receive parameter information sent by the first terminal; wherein the parameter information includes live streaming parameters and hardware parameters; The live streaming parameters are input into the target model to obtain the initial encoding parameters output by the target model. The live streaming parameters include: live streaming type and device type. The device type includes one of the following: smart terminal, desktop computer, or third-party server. The target model is obtained by training using the following method: selecting one or more sample data sets containing the same historical live streaming parameters or the same historical encoding parameters; obtaining the target sample data with the highest historical quality score from the selected one or more sample data sets; inputting the historical live streaming parameters from the target sample data into the initial model to obtain the prediction information output by the initial model; obtaining a loss function based on the prediction information and the historical encoding parameters from the target sample data; and updating the initial model based on the loss function through backpropagation to obtain the target model. Based on the hardware parameters and the initial encoding parameters, the target encoding parameters are determined; The target encoding parameters are sent to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream. The step of determining the target encoding parameters based on the hardware parameters and the initial encoding parameters includes: determining candidate encoding-related parameters based on the hardware parameters; determining whether the content of the candidate encoding-related parameters is the same as that of the parameters of the same type in the initial encoding parameters; and adjusting the initial encoding parameters based on the candidate encoding-related parameters if the content of the parameters of the same type in the candidate encoding-related parameters is different from that of the initial encoding parameters to obtain the target encoding parameters. The step of determining candidate coding-related parameters based on the hardware parameters includes: determining the hardware score of the first terminal based on the hardware parameters; and determining the candidate coding-related parameters corresponding to the hardware score based on a preset first correspondence relationship. The first correspondence relationship includes: candidate coding-related parameters corresponding to each candidate hardware score range in one or more candidate hardware score ranges.

2. The method according to claim 1, wherein the sample data comprises: Historical live streaming parameters corresponding to the historical multimedia data stream, historical encoding parameters corresponding to the historical multimedia data stream, and historical quality scores corresponding to the historical multimedia data stream.

3. The method according to claim 1, wherein, The step of determining the target encoding parameters based on the hardware parameters and the initial encoding parameters includes: Based on the hardware parameters, determine the relevant parameters for the candidate encoding; If the candidate encoding parameters are the same as the initial encoding parameters, the initial encoding parameters are used as the target encoding parameters.

4. The method according to any one of claims 1-3, wherein, After sending the target encoding parameters to the first terminal, the method further includes: The current network parameters of the first terminal are obtained in real time; Based on the current network parameters of the first terminal and the historical network parameters of the first terminal, the network change status of the first terminal is determined; If it is determined that the target coding parameters need to be adjusted based on the network change status of the first terminal, the adjusted target coding parameters are determined. The adjusted target encoding parameters are sent to the first terminal.

5. The method according to claim 4, further comprising: Based on the live streaming parameters and the hardware parameters, the current transcoding parameters corresponding to the first terminal are determined.

6. The method according to claim 5, further comprising: The current transcoding parameters corresponding to the first terminal are sent to the transcoding server; wherein, the current transcoding parameters corresponding to the first terminal are used by the transcoding server to transcode the encoded multimedia data stream uploaded by the first terminal to obtain multimedia data streams of each of N bitrates; N is an integer greater than or equal to 1.

7. The method according to claim 5, further comprising: Upon receiving the encoded multimedia data stream uploaded by the first terminal, the encoded multimedia data stream is transcoded based on the current transcoding parameters corresponding to the first terminal to obtain multimedia data streams for each of N bitrates; N is an integer greater than or equal to 1. The multimedia data stream of each of the N bitrates is sent to the edge server.

8. A data flow control system, comprising: A data flow control device is used to receive parameter information sent by a first terminal; wherein the parameter information includes live streaming parameters and hardware parameters; inputting the live streaming parameters into a target model to obtain initial encoding parameters output by the target model, wherein the live streaming parameters include: live streaming type and device type, wherein the device type includes one of the following: smart terminal, desktop computer, third-party server; determining target encoding parameters based on the hardware parameters and the initial encoding parameters; sending the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain an encoded multimedia data stream; wherein the target model is obtained by training using the following training method: selecting one or more sample data containing the same historical live streaming parameters or the same historical encoding parameters; obtaining the target sample data with the highest historical quality score from the selected one or more sample data; inputting the historical live streaming parameters in the target sample data into the initial model to obtain prediction information output by the initial model; obtaining a loss function based on the prediction information and the historical encoding parameters in the target sample data; updating the initial model based on the loss function through backpropagation to obtain the target model; A first terminal is configured to send the parameter information and receive the target encoded parameters. Specifically, the data flow control device is used to determine candidate encoding-related parameters based on the hardware parameters; determine whether the content of the candidate encoding-related parameters is the same as that of the parameter of the same type in the initial encoding parameters; and, if the content of the parameter of the same type in the candidate encoding-related parameters is different from that of the initial encoding parameters, adjust the initial encoding parameters based on the candidate encoding-related parameters to obtain the target encoding parameters. The step of determining candidate coding-related parameters based on the hardware parameters includes: determining the hardware score of the first terminal based on the hardware parameters; and determining the candidate coding-related parameters corresponding to the hardware score based on a preset first correspondence relationship. The first correspondence relationship includes: candidate coding-related parameters corresponding to each candidate hardware score range in one or more candidate hardware score ranges.

9. The system according to claim 8, wherein, The sample data includes: historical live streaming parameters corresponding to historical multimedia data streams, historical encoding parameters corresponding to historical multimedia data streams, and historical quality scores corresponding to historical multimedia data streams.

10. The system according to claim 8, wherein, The data flow control device is used to determine candidate encoding related parameters based on the hardware parameters; if the candidate encoding related parameters are the same as the initial encoding parameters, the initial encoding parameters are used as the target encoding parameters.

11. The system according to any one of claims 8-10, wherein, The data flow control device is used to acquire the current network parameters of the first terminal in real time; determine the network change status of the first terminal based on the current network parameters and the historical network parameters of the first terminal; and determine the adjusted target encoding parameters if it is determined that the target encoding parameters need to be adjusted based on the network change status of the first terminal. Send the adjusted target encoding parameters to the first terminal; The first terminal is used to receive the adjusted target encoding parameters.

12. The system according to claim 11, wherein, The data flow control device is used to determine the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

13. The system according to claim 12, wherein, The system also includes: A transcoding server is used to receive the current transcoding parameters corresponding to the first terminal; the current transcoding parameters corresponding to the first terminal are used to transcode the encoded multimedia data stream uploaded by the first terminal to obtain a multimedia data stream corresponding to each of N bitrates, where N is an integer greater than or equal to 1. The data flow control device is used to send the current transcoding parameters corresponding to the first terminal.

14. The system according to claim 13, wherein, The system also includes: An edge server is used to receive multimedia data streams corresponding to each of N bitrates. The second terminal is used to obtain the multimedia data stream with a matching bit rate from the multimedia data stream corresponding to each of the N bit rates. The first terminal is configured to, upon acquiring the original multimedia data stream, encode the original multimedia data stream based on current encoding parameters to obtain an encoded multimedia data stream; and send the encoded multimedia data stream to the transcoding server; wherein the current encoding parameters are the target encoding parameters or the adjusted target encoding parameters; The transcoding server is configured to, upon receiving the encoded multimedia data stream, transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain a multimedia data stream corresponding to each of the N bitrates; and send the multimedia data stream corresponding to each of the N bitrates.

15. The system according to claim 12, wherein, The system also includes: An edge server is used to receive multimedia data streams corresponding to each of N bitrates. The second terminal is used to obtain the multimedia data stream with a matching bit rate from the multimedia data stream corresponding to each of the N bit rates. The first terminal is configured to, upon acquiring an original multimedia data stream, encode the original multimedia data stream based on current encoding parameters to obtain an encoded multimedia data stream; and send the encoded multimedia data stream to the data stream control device; wherein the current encoding parameters are the target encoding parameters or the adjusted target encoding parameters; The data stream control device is configured to, upon receiving the encoded multimedia data stream, transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal to obtain a multimedia data stream corresponding to each of the N bitrates; and send the multimedia data stream corresponding to each of the N bitrates.

16. A data flow control device, comprising: A receiving module is used to receive parameter information sent by a first terminal; wherein the parameter information includes live streaming parameters and hardware parameters; The processing module is used to input the live streaming parameters into the target model to obtain the initial encoding parameters output by the target model. The live streaming parameters include: live streaming type and device type, where the device type includes one of the following: smart terminal, desktop computer, or third-party server. Based on the hardware parameters and the initial encoding parameters, target encoding parameters are determined. The target model is obtained by training using the following method: selecting one or more sample data sets containing the same historical live streaming parameters or the same historical encoding parameters; obtaining the target sample data with the highest historical quality score from the selected one or more sample data sets; inputting the historical live streaming parameters from the target sample data into the initial model to obtain the prediction information output by the initial model; obtaining a loss function based on the prediction information and the historical encoding parameters from the target sample data; and updating the initial model based on the loss function through backpropagation to obtain the target model. The sending module is used to send the target encoding parameters to the first terminal; wherein the target encoding parameters are used to encode the original multimedia data stream to obtain the encoded multimedia data stream; Specifically, the processing module is used to determine candidate encoding-related parameters based on the hardware parameters; determine whether the content of the candidate encoding-related parameters is the same as that of the parameters of the same type in the initial encoding parameters; and if the content of the parameters of the same type in the candidate encoding-related parameters is different from that in the initial encoding parameters, adjust the initial encoding parameters based on the candidate encoding-related parameters to obtain the target encoding parameters. The step of determining candidate coding-related parameters based on the hardware parameters includes: determining the hardware score of the first terminal based on the hardware parameters; and determining the candidate coding-related parameters corresponding to the hardware score based on a preset first correspondence relationship. The first correspondence relationship includes: candidate coding-related parameters corresponding to each candidate hardware score range in one or more candidate hardware score ranges.

17. The apparatus according to claim 16, wherein, The sample data includes: historical live streaming parameters corresponding to historical multimedia data streams, historical encoding parameters corresponding to historical multimedia data streams, and historical quality scores corresponding to historical multimedia data streams.

18. The apparatus according to claim 16, wherein, The processing module is used to determine candidate encoding-related parameters based on the hardware parameters; if the candidate encoding-related parameters are the same as the initial encoding parameters, the initial encoding parameters are used as the target encoding parameters.

19. The apparatus according to any one of claims 16-18, further comprising: The adjustment module is used to determine the network change status of the first terminal based on the current network parameters of the first terminal and the historical network parameters of the first terminal. If it is determined that the target coding parameters need to be adjusted based on the network change status of the first terminal, the adjusted target coding parameters are determined. The receiving module is used to acquire the current network parameters of the first terminal in real time; The sending module is used to send the adjusted target encoding parameters to the first terminal.

20. The apparatus according to claim 19, wherein, The processing module is used to determine the current transcoding parameters corresponding to the first terminal based on the live streaming parameters and the hardware parameters.

21. The apparatus according to claim 20, wherein, The sending module is used to send the current transcoding parameters corresponding to the first terminal to the transcoding server; wherein, the current transcoding parameters corresponding to the first terminal are used by the transcoding server to transcode the encoded multimedia data stream uploaded by the first terminal to obtain multimedia data streams of each of N bitrates; N is an integer greater than or equal to 1.

22. The apparatus of claim 20, further comprising: The transcoding module is used to transcode the encoded multimedia data stream based on the current transcoding parameters corresponding to the first terminal when receiving the encoded multimedia data stream uploaded by the first terminal, to obtain a multimedia data stream of each of N bitrates; N is an integer greater than or equal to 1. The sending module is used to send the multimedia data stream of each of the N bitrates to the edge server.

23. An electronic device, comprising: At least one processor; as well as A memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-7.

24. A non-transitory computer-readable storage medium storing computer instructions, wherein, The computer instructions are used to cause the computer to perform the method according to any one of claims 1-7.

25. A computer program product comprising a computer program that, when executed by a processor, implements the method according to any one of claims 1-7.

Images