Multimedia processing method and device in real-time communication, and storage medium

Abstract

Embodiments of the present disclosure provide a multimedia processing method and device in real-time communication, and a storage medium. The multimedia processing method in real-time communication takes real-time performance parameters of a terminal device and a real-time load condition of a target multimedia processing process into consideration when determining whether to adjust multimedia processing parameters of the target multimedia processing process in real-time communication, thereby improving accuracy in determining the adjustment timing, and avoiding severe impacts on real-time communication caused by misjudgment of the adjustment timing. Moreover, after candidate multimedia processing parameters are obtained by adjustment, a quality-of-service prediction model is first used for verification, and the candidate multimedia processing parameters are applied to the target multimedia processing process only after the suitability of the parameters has been confirmed, thereby avoiding frequent parameter modifications to the target multimedia processing process, ensuring rapid and efficient parameter adjustment in the target multimedia processing process, and ensuring stability of real-time communication.

Description

Multimedia processing method, device and storage medium in real-time communication

[0001] This application claims priority to Chinese Patent Application No. 202410339375.4 filed on March 22, 2024, and the contents of the above-mentioned Chinese patent application disclosure are hereby incorporated by reference in their entirety as a part of this application. Technical Field

[0002] The embodiments of the present disclosure relate to a multimedia processing method, device, and storage medium in real-time communication. Background Art

[0003] Real-time Communications (RTC) is a technology used to enable real-time audio, video, and data transmission. It allows users to communicate in real time over the internet, including voice and video calls, and real-time data transmission. With the development of the times, RTC has become a new source of productivity in the digital economy. Digital scenarios based on RTC technology, such as e-commerce live streaming, cloud gaming, remote conferencing, and online learning, are booming. RTC's use cases are becoming increasingly diverse, and its functionality is also growing.

[0004] However, the increasing number of special effects and ever-higher resolution and frame rate requirements are placing an increasing performance load on end-devices. Under these high performance loads, various issues can easily arise, such as frame drops, lags, excessive power consumption, and excessive heat, all of which severely impact the user experience. Addressing these issues requires the audio and video engine to adaptively adjust parameters based on the end-device's performance, ensuring both smoothness and clarity while ensuring the normal operation of basic functions.

[0005] However, the timing of parameter adjustment during the RTC process is prone to misjudgment, and multiple adjustments are required during the adjustment process, which has low adjustment efficiency, resulting in freezes and delays, affecting the stability of RTC communication quality. Summary of the Invention

[0006] The embodiments of the present disclosure provide a multimedia processing method, device, and storage medium in real-time communication to ensure fast, efficient, and accurate parameter adjustment in real-time communication and ensure the stability of real-time communication.

[0007] In a first aspect, an embodiment of the present disclosure provides a multimedia processing method in real-time communication, applied to a terminal device, the method comprising:

[0008] Acquiring real-time performance parameters of the terminal device in real-time communication, determining initial multimedia processing parameters of at least one target multimedia processing process in the real-time communication based on the real-time performance parameters, and applying the initial multimedia processing parameters to the target multimedia processing process;

[0009] obtaining a real-time load condition of the target multimedia processing process; if the real-time load condition exceeds a preset load threshold, adjusting the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters;

[0010] Predicting a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied using a service quality prediction model based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device;

[0011] If it is determined that the service quality indicator prediction value meets the preset service quality requirement, the candidate multimedia processing parameter is applied to the target multimedia processing process.

[0012] In a second aspect, an embodiment of the present disclosure provides a multimedia processing device for real-time communication, including:

[0013] A collection unit, used to obtain real-time performance parameters of terminal devices in real-time communication;

[0014] an initial parameter determination unit, configured to determine initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters, and apply the initial multimedia processing parameters to the target multimedia processing process;

[0015] a parameter adjustment unit, configured to obtain a real-time load condition of the target multimedia processing process; and if the real-time load condition exceeds a preset load threshold, adjust the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters;

[0016] a prediction unit, configured to use a quality of service prediction model to predict a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied, based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device;

[0017] An execution unit is configured to apply the candidate multimedia processing parameters to the target multimedia processing process if it is determined that the service quality indicator prediction value meets the preset service quality requirement.

[0018] In a third aspect, an embodiment of the present disclosure provides an electronic device, including: at least one processor and a memory;

[0019] The memory stores computer-executable instructions;

[0020] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect.

[0021] In a fourth aspect, an embodiment of the present disclosure provides a computer-readable storage medium, in which computer execution instructions are stored. When a processor executes the computer execution instructions, the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect is implemented.

[0022] In a fifth aspect, an embodiment of the present disclosure provides a computer program product, comprising computer-executable instructions. When a processor executes the computer-executable instructions, the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect is implemented. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present disclosure, a brief introduction will be given below to the drawings required for use in the description of the embodiments. Obviously, the drawings described below are some embodiments of the present disclosure. For ordinary technicians in this field, other drawings can be obtained based on these drawings without any creative work.

[0024] FIG1 is an example diagram of a multimedia processing method in real-time communication provided by an embodiment of the present disclosure;

[0025] FIG2 is a flow chart of a multimedia processing method in real-time communication provided by an embodiment of the present disclosure;

[0026] FIG3 is a schematic diagram of the structure of a service quality prediction model provided by an embodiment of the present disclosure;

[0027] FIG4 is a structural block diagram of a multimedia processing device in real-time communication provided by an embodiment of the present disclosure; and

[0028] FIG5 is a schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present disclosure. DETAILED DESCRIPTION

[0029] To make the objectives, technical solutions, and advantages of the embodiments of the present disclosure more clear, the technical solutions in the embodiments of the present disclosure will be clearly and completely described below in conjunction with the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present disclosure, not all of the embodiments. All other embodiments obtained by ordinary technicians in this field based on the embodiments of the present disclosure without making any creative efforts shall fall within the scope of protection of the present disclosure.

[0030] In RTC (Real-time Communications), the increasing number of special effects and ever-higher resolution and frame rate requirements place an ever-increasing performance load on terminal devices. Under these high performance loads, various issues can easily arise, such as frame drops, lags, excessive power consumption, and excessive heat generation, all of which severely impact the user experience. Addressing these issues requires the audio and video engine to adaptively adjust parameters based on the performance of the terminal device, ensuring a balance between smoothness and clarity while ensuring the normal operation of basic functions.

[0031] However, most mainstream RTC audio and video engines draw on and reference WebRTC, and have a certain degree of performance adaptability in encoding processing. WebRTC calculates the encoder occupancy rate based on the input and output frame rates of the encoding node. When the encoder occupancy rate exceeds the upper occupancy rate threshold, parameter downgrade is triggered, and when the encoder occupancy rate falls below the lower occupancy rate threshold, parameter upgrade is triggered. Depending on the quality preference, the downgrade and upgrade operations will gradually adjust the resolution and / or frame rate proportionally until the encoder occupancy rate is between the upper and lower occupancy rate thresholds. However, this parameter adjustment method has the following problems:

[0032] 1) The judgment of encoder parameter upgrade or downgrade depends solely on the comparison between the encoder occupancy rate and the threshold, and is not directly related to the service quality indicator QoS. This can easily lead to misjudgment of the timing of parameter adjustment. For example, although the encoder occupancy rate is higher than the upper occupancy rate threshold, the service quality indicator QoS is good, with no lag or delay. In this case, parameter downgrade is triggered, reducing the resolution and / or frame rate, affecting the clarity. Alternatively, although the encoder occupancy rate is lower than the lower occupancy rate threshold, the service quality indicator QoS is poor, with lag and delay. In this case, parameter upgrade is triggered, which may aggravate the lag and delay, affecting the smoothness.

[0033] 2) The encoder parameter upgrade and downgrade is performed step by step, which prolongs the adjustment time. Multiple parameter adjustments cause lag and delay in the entire link, which directly affects the stability of the call quality of real-time communication. In addition, if the adjustment amplitude is small each time, the number of adjustments increases. If the adjustment amplitude is large each time, the parameter downgrade may be too large, causing the encoder occupancy rate to fall below the lower occupancy rate threshold, triggering a parameter upgrade. During the upgrade, the parameter upgrade may be too large, causing the encoder occupancy rate to rise above the upper occupancy rate threshold, triggering a parameter downgrade again. This leads to repeated parameter upgrades and downgrades, low adjustment efficiency, and difficulty in quickly adjusting to the appropriate parameters.

[0034] 3) In addition, in addition to the encoder, other processing processes in real-time communication, such as acquisition, decoding, rendering, image pre- and post-processing, etc., are prone to problems due to performance load, but most audio and video engines do not have the ability to monitor and adjust other processing processes in real-time communication.

[0035] In order to solve the above technical problems, the present disclosure provides a multimedia processing method in real-time communication, so as to at least accurately judge the timing of parameter adjustment in real-time communication, more reasonably select parameters, reduce frequent changes in parameters, achieve fast and efficient adjustment, and ensure the stability of real-time communication. In order to achieve the above purpose, the present disclosure considers the real-time performance parameters of the terminal device and the real-time load of the target multimedia processing process in the judgment of whether to adjust the multimedia processing parameters of the target multimedia processing process in real-time communication, improves the accuracy of judging the adjustment timing, avoids the misjudgment of the adjustment timing and causes serious impact on real-time communication; and after the candidate multimedia processing parameters are adjusted, a service quality prediction model is used to verify them first, and after determining the appropriate candidate multimedia processing parameters, they are applied to the target multimedia processing process, avoiding frequent parameter modifications of the target multimedia processing process, ensuring fast and efficient adjustment of the parameters in the target multimedia processing process, and ensuring the stability of real-time communication.

[0036] Specifically, by obtaining real-time performance parameters of a terminal device in real-time communication, initial multimedia processing parameters of at least one target multimedia processing process in real-time communication are determined based on the real-time performance parameters, and the initial multimedia processing parameters are applied to the target multimedia processing process; the real-time load condition of the target multimedia processing process is obtained; if the real-time load condition exceeds a preset load threshold, the initial multimedia processing parameters are adjusted based on the real-time service quality indicators of the target multimedia processing process to obtain candidate multimedia processing parameters; based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process and the real-time performance parameters of the terminal device, a service quality prediction model is used to predict the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied; if it is determined that the service quality indicator prediction value meets the preset service quality requirement, the candidate multimedia processing parameters are applied to the target multimedia processing process.

[0037] The multimedia processing method in real-time communication provided by the present disclosure is applied to a terminal device, and its process framework is shown in FIG1 , which obtains the real-time performance parameters of the terminal device in real-time communication, including but not limited to CPU usage, GPU usage, temperature, and power, and transmits the real-time performance parameters of the terminal device to a performance control center module in the terminal device, and the performance control center module determines the initial multimedia processing parameters of at least one target multimedia processing process in real-time communication based on the real-time performance parameters, wherein the target multimedia processing process may include a multimedia processing process in an uplink phase and / or a multimedia processing process in a downlink phase, and sends the initial multimedia processing parameters to the uplink performance controller module and / or the downlink performance controller module, and the uplink performance controller module and / or the downlink performance controller module applies the initial multimedia processing parameters to the target multimedia processing process, and the target multimedia processing process can be obtained. The real-time load condition of the target multimedia processing process is collected; if the real-time load condition exceeds the preset load threshold, the real-time service quality index of the target multimedia processing process is collected, and the uplink local performance controller module and / or the downlink local performance controller module respectively adjust the initial multimedia processing parameters of each target multimedia processing process according to the real-time service quality index of each target multimedia processing process to obtain candidate multimedia processing parameters, and based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process and the real-time performance parameters of the terminal device, a service quality prediction model is used to predict the service quality index prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied. If it is determined that the service quality index prediction value meets the preset service quality requirements, the candidate multimedia processing parameters are applied to the target multimedia processing process to achieve local performance control of the target multimedia processing process. In the corresponding downlink stage, when the candidate multimedia processing parameters are applied to the target multimedia processing process, the to-be-processed multimedia data that meets the candidate multimedia processing parameters can be obtained from the server through subscription information update.

[0038] It should be noted that the user information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this disclosure are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0039] The multimedia processing method in real-time communication disclosed in the present invention will be described in detail below with reference to specific embodiments.

[0040] Referring to FIG2 , FIG2 is a flow chart of a multimedia processing method in real-time communication provided by an embodiment of the present disclosure. The method of this embodiment can be applied in a terminal device or a server. The multimedia processing method in real-time communication includes:

[0041] S201. Acquire real-time performance parameters of a terminal device in real-time communication, determine initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters, and apply the initial multimedia processing parameters to the target multimedia processing process.

[0042] In this embodiment, during RTC real-time communication on a terminal device, real-time performance parameters of the terminal device can be obtained, including but not limited to CPU usage, GPU usage, power consumption, temperature, etc. Real-time communication can include but is not limited to real-time communication in scenarios such as e-commerce live streaming, cloud gaming, remote conferencing, and online teaching.

[0043] When a terminal device is under different real-time performance parameters, in order to ensure the smoothness and clarity of real-time communication, the real-time performance parameters can be used to determine the parameters of at least one target multimedia processing process in the real-time communication, where the target multimedia processing process may include a multimedia processing process in the uplink phase (stream push) and / or a multimedia processing process in the downlink phase (stream pull). The multimedia processing process in the uplink phase includes but is not limited to encoding, special effects such as beauty, noise reduction, etc., and the multimedia processing process in the downlink phase includes but is not limited to decoding, super-resolution, rendering, etc. In this embodiment, initial multimedia processing parameters can be pre-configured for at least one target multimedia processing process in real-time communication of the terminal device under different real-time performance parameters. The types of initial multimedia processing parameters of the target multimedia processing process may be different. For example, the types of initial multimedia processing parameters in the encoding process and the decoding process may be resolution and / or frame rate, while the types of initial multimedia processing parameters for special effects such as beauty, noise reduction, and super-resolution processes may be whether to start, the selection of algorithm complexity, etc. Further, the initial multimedia processing parameters of at least one target multimedia processing process in the real-time communication can be determined based on the real-time performance parameters of the terminal device in the real-time communication, and the initial multimedia processing parameters can be applied to the target multimedia processing process.

[0044] Optionally, global parameter configuration can be performed in advance for real-time communications of terminal devices under different real-time performance parameters to balance the load between various multimedia processing processes, thereby obtaining a preset multimedia processing parameter set, wherein the preset multimedia processing parameter set includes initial multimedia processing parameters of multiple multimedia processing processes in real-time communications of terminal devices under certain real-time performance parameters, wherein the multiple multimedia processing processes may be all or part of the multimedia processing processes in real-time communications, especially including the above-mentioned target multimedia processing processes, and in order to maximize the performance of the terminal device and improve fluency and clarity, the preset multimedia processing parameter set includes multimedia processing parameter thresholds of multiple multimedia processing processes in real-time communications (that is, the limit values ​​that can maximize the performance of the terminal device and improve fluency and clarity).

[0045] Furthermore, a set of preset multimedia processing parameter sets may be determined based on the real-time performance parameters of the terminal device in real-time communication, and each multimedia processing parameter included in the preset multimedia processing parameter set may be applied to the corresponding multimedia processing process.

[0046] S202. Obtain the real-time load condition of the target multimedia processing process; if the real-time load condition exceeds a preset load threshold, adjust the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters.

[0047] In this embodiment, the real-time load of the target multimedia processing process can be obtained. For example, for an encoding process, the real-time load can be the encoder occupancy rate. If the real-time load of the target multimedia processing process exceeds a preset load threshold, a real-time quality of service indicator of the target multimedia processing process, such as QoS (Quality of Service), can be further obtained. This can be used to reflect freezes or delays. Based on the real-time quality of service indicator of the target multimedia processing process, it can be determined whether to adjust the initial multimedia processing parameters, how to adjust them, and thus obtain candidate multimedia processing parameters after adjustment.

[0048] Optionally, if the real-time load of the target multimedia processing process is greater than the preset load threshold upper limit, it is determined whether the real-time service quality index of the target multimedia processing process meets the preset service quality requirements; if it does not meet the preset service quality requirements, the initial multimedia processing parameters are adjusted to reduce the real-time load of the target multimedia processing process to obtain the candidate multimedia processing parameters.

[0049] In this embodiment, if the real-time load of the target multimedia processing process is greater than the preset load threshold upper limit, it means that the load of the target multimedia processing process is too large. In theory, the load of the target multimedia processing process needs to be reduced. However, if the jamming or delay of the target multimedia processing process is within an acceptable range or is not affected, the load of the target multimedia processing process can be kept unchanged. However, if the jamming or delay of the target multimedia processing process exceeds the acceptable range or is severely affected, the load of the target multimedia processing process needs to be reduced. That is, it can be determined whether the real-time service quality indicator of the target multimedia processing process meets the preset service quality requirement. If it does not meet the preset service quality requirement, the initial multimedia processing parameters are adjusted to reduce the real-time load of the target multimedia processing process to obtain candidate multimedia processing parameters. For example, if the encoder occupancy rate during the encoding process is higher than the preset encoder occupancy rate upper limit, it is determined whether there is severe jamming during the encoding process (for example, the jamming duration exceeds the preset duration) based on the QoS of the encoding process. If there is severe jamming, the resolution and / or frame rate are reduced. If there is no severe jamming, the current resolution and frame rate can be maintained, and the adjustment process can be terminated.

[0050] Optionally, if the real-time load condition of the target multimedia processing process is less than the preset load threshold lower limit, it is determined whether the real-time service quality indicator of the target multimedia processing process continues to meet the preset service quality requirements for a preset period of time; if it continues to meet the preset service quality requirements for a preset period of time, the initial multimedia processing parameters are adjusted to improve the real-time load condition of the target multimedia processing process, and the candidate multimedia processing parameters are obtained.

[0051] In this embodiment, if the real-time load of the target multimedia processing process is less than the preset load threshold lower limit, it means that the target multimedia processing process is currently under a small load and can handle a larger load. In theory, the load of the target multimedia processing process can be increased. However, if the target multimedia processing process is experiencing a jam or delay that exceeds an acceptable range, even if the current load is small, the load of the target multimedia processing process cannot be increased. However, if the jam or delay of the target multimedia processing process is within an acceptable range or is not affected, the load of the target multimedia processing process can be increased. That is, it can be determined whether the real-time quality of service indicator of the target multimedia processing process continuously meets the preset quality of service requirement for a preset time period. If the preset quality of service requirement is continuously met for a preset time period, the initial multimedia processing parameters are adjusted to increase the real-time load of the target multimedia processing process, thereby obtaining candidate multimedia processing parameters. For example, during the encoding process, if the encoder occupancy rate is lower than the preset encoder occupancy rate lower limit, it is determined based on the QoS of the encoding process whether there is no jamming during the encoding process for a preset time period (e.g., 100 seconds). If so, the resolution and / or frame rate can be increased. If not, the current resolution and frame rate can be maintained, and the adjustment process can terminate.

[0052] In the above process, the multimedia processing parameters are adjusted while considering the real-time service quality index of the target multimedia processing process, which is beneficial to improving the service quality of the target multimedia processing process and avoiding misjudgment of the adjustment timing.

[0053] S203. Based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device, a service quality prediction model is used to predict a predicted value of a service quality indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied.

[0054] In this embodiment, after obtaining the candidate multimedia processing parameters, in order to efficiently verify whether the candidate multimedia processing parameters are appropriate, a model can be used for prediction in this embodiment. Specifically, a service quality prediction model can be pre-trained for the target multimedia processing process. Its input is the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process. The output is the predicted value of the service quality indicator after the candidate multimedia processing parameters are applied to the target multimedia processing process. Among them, the attribute information of the target multimedia processing process includes but is not limited to the business scenario, protocol, algorithm, system platform, etc. of the target multimedia processing process, and the real-time performance parameters of the terminal device include CPU usage, GPU usage, temperature, power, etc. The service quality prediction model can adopt any feasible known model, such as a deep learning model, etc. The specific training process is not limited in this embodiment. In this embodiment, the training process of the service quality prediction model can be completed on the terminal device, or it can also be completed on other devices such as servers, and then the trained model is deployed on the terminal device.

[0055] S204: If it is determined that the service quality indicator prediction value meets the preset service quality requirement, the candidate multimedia processing parameter is applied to the target multimedia processing process.

[0056] In this embodiment, after predicting the service quality indicator prediction value of the target multimedia processing process when applying the candidate multimedia processing parameters through the service quality prediction model, it can be determined whether the service quality indicator prediction value meets the preset service quality requirements. If the service quality indicator prediction value meets the preset service quality requirements, it means that the candidate multimedia processing parameters are appropriate and the candidate multimedia processing parameters can be directly applied to the target multimedia processing process.

[0057] If the predicted value of the service quality indicator does not meet the preset service quality requirements, it means that the candidate multimedia processing parameters are inappropriate. The candidate multimedia processing parameters can be readjusted to obtain new candidate multimedia processing parameters. The service quality prediction model is further used to predict the predicted value of the service quality indicator of the target multimedia processing process when the new candidate multimedia processing parameters are applied based on the new candidate multimedia processing parameters, the attribute information of the target multimedia processing process and the real-time performance parameters of the terminal device, and the service quality indicator prediction value is re-judged to determine whether the preset service quality requirements are met. The above process is repeated until the predicted value of the service quality indicator meets the preset service quality requirements, and the candidate multimedia processing parameters that finally meet the preset service quality requirements are applied to the target multimedia processing process.

[0058] Among them, candidate multimedia processing parameters of different gears can be configured for the target multimedia processing process. When the initial multimedia processing parameters are adjusted in S202 and the candidate multimedia processing parameters are obtained, the candidate multimedia processing parameters can be selected by adjusting the gear. When the service quality indicator prediction value does not meet the preset service quality requirements and the candidate multimedia processing parameters are readjusted, the candidate multimedia processing parameters can also be reselected by adjusting the gear, such as raising the gear or lowering the gear.

[0059] The multimedia processing method in real-time communication provided by this embodiment obtains real-time performance parameters of a terminal device in real-time communication, determines initial multimedia processing parameters of at least one target multimedia processing process in real-time communication based on the real-time performance parameters, and applies the initial multimedia processing parameters to the target multimedia processing process; obtains the real-time load status of the target multimedia processing process; if the real-time load status exceeds a preset load threshold, adjusts the initial multimedia processing parameters based on the real-time service quality indicators of the target multimedia processing process to obtain candidate multimedia processing parameters; uses a service quality prediction model to predict a predicted value of the service quality indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied based on the candidate multimedia processing parameters, attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device; and if it is determined that the predicted value of the service quality indicator meets the preset service quality requirements, applies the candidate multimedia processing parameters to the target multimedia processing process. In this embodiment, the judgment of whether to adjust the multimedia processing parameters of the target multimedia processing process in real-time communication takes into account the real-time performance parameters of the terminal device and the real-time load of the target multimedia processing process, thereby improving the accuracy of judging the adjustment timing and avoiding misjudgment of the adjustment timing that may cause serious impact on real-time communication. After the candidate multimedia processing parameters are adjusted, a service quality prediction model is used to verify them first. After determining the appropriate candidate multimedia processing parameters, they are applied to the target multimedia processing process, thereby avoiding frequent modification of the parameters of the target multimedia processing process, ensuring fast and efficient adjustment of the parameters in the target multimedia processing process, and ensuring the stability of real-time communication.

[0060] Based on any of the above embodiments, when using the service quality prediction model for prediction, the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process may contain text type data or numerical type data. Therefore, the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process can be divided into text type input data and numerical type input data. Taking the encoding process as an example, the text type input data may include:

[0061] Business scenario tags: live streaming, multi-person live broadcast, video conferencing, cloud gaming;

[0062] Coding protocol type: video compression standard, such as AVC, HEVC, VP8, etc.

[0063] Encoder type: the implementation method of video compression encoding, such as QSV (Windows platform hardware encoder), OpenH264 (software encoder), MediaCodec (Android hardware encoder), VideoToolbox (iOS, Mac hardware encoder);

[0064] Encoder preset parameters: A set of preset parameters used to control the image quality and compression efficiency during the video compression process. Common gears include Slow, Medium, Fast, and Ultrafast. Generally speaking, Slow encoding is the slowest but has the highest image quality; Ultrafast encoding is the fastest but has the worst image quality.

[0065] System platform: IOS, Android, Windows, MacOS;

[0066] Numeric input data can include:

[0067] Encoding resolution, encoding frame rate, CPU usage, battery temperature, etc.

[0068] When using the service quality prediction model to predict the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied, the method may specifically include:

[0069] The text type input data and the numerical type input data are input into the service quality prediction model, and features are extracted from the text type input data and the numerical type input data respectively. The features are learned using a multi-head self-attention layer and a fully connected layer, and the service quality indicator prediction value is output through the output layer as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied.

[0070] In this embodiment, in the service quality prediction model, features can be extracted separately for text type input data and numerical type input data. Optionally, the architecture of the service quality prediction model can be as shown in Figure 3, using an embedding layer (Embedding) to vectorize the text type input data, and then performing linear transformation through a linear layer (Linear) to obtain features corresponding to the text type input data; in addition, a linear layer (Linear) can be used to perform linear transformation on the numerical type input data to obtain features corresponding to the numerical type input data; further, the features corresponding to the text type input data and the features corresponding to the numerical type input data are input into a multi-head self-attention layer for processing, and then the processing results are input into a fully connected layer, and feature learning is achieved through the multi-head self-attention layer and the fully connected layer. Finally, the processing results are input into the output layer for prediction, and the service quality indicator prediction value is calculated through a linear function (Linear Function) as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied. Through feature learning of multi-head self-attention layers and fully connected layers, the features of complex text-type input data and numerical-type input data can be better processed, processing efficiency and feature learning accuracy can be improved, and ultimately the accuracy of predicted service quality indicator predictions can be improved.

[0071] In an optional embodiment, when determining the initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters in S201, for certain target multimedia processing processes, for example, some non-essential multimedia processing processes, such as special effects such as beauty, noise reduction, super-resolution, etc., processing processes, it can be determined whether any target multimedia processing process is started according to the real-time performance parameters; or a target algorithm is selected from a set of alternative algorithms for any target multimedia processing process according to the real-time performance parameters, wherein the set of alternative algorithms includes alternative algorithms of different complexities. Specifically, when one or more of the CPU usage rate, GPU usage rate, and temperature are high and / or the battery power is low, one or more of the special effects such as beauty, noise reduction, super-resolution, etc., can be turned off, or an alternative algorithm with lower complexity can be selected to implement the configuration of the initial multimedia processing parameters and ensure the normal use of important functions in real-time communication; when one or more of the CPU usage rate, GPU usage rate, and temperature are low and / or the battery power is high, one or more of the special effects such as beauty, noise reduction, super-resolution, etc., can be turned on, or an alternative algorithm with higher complexity can be selected to improve video quality and enhance user experience.

[0072] Optionally, in S202, if the real-time load of the target multimedia processing process exceeds a preset load threshold, the initial multimedia processing parameters are adjusted according to the real-time service quality index of the target multimedia processing process. When the candidate multimedia processing parameters are obtained, for certain target multimedia processing processes, such as some non-essential multimedia processing processes, such as special effects such as beauty, noise reduction, super-resolution and other processing processes, it is possible to adjust whether the target multimedia processing process is started, or select a target algorithm of appropriate complexity from a set of alternative algorithms for any target multimedia processing process according to the real-time performance parameters. Specifically, if the real-time load of the target multimedia processing process is greater than the preset load threshold upper limit, it is determined whether the real-time service quality index of the target multimedia processing process meets the requirements. If the preset service quality requirements are not met, the target multimedia processing process is closed, or a target algorithm with a lower complexity than the current algorithm is selected from the set of alternative algorithms, that is, the algorithm complexity is reduced to ensure the normal use of important functions in real-time communication; or, if the real-time load of the target multimedia processing process is less than the preset load threshold lower limit, it is determined whether the real-time service quality index of the target multimedia processing process continues to meet the preset service quality requirements for a preset time. If the preset service quality requirements are continuously met for a preset time, the target multimedia processing process is kept open. In addition, a target algorithm with a higher complexity than the current algorithm can be selected from the set of alternative algorithms, that is, the algorithm complexity is increased, the video quality is improved, and the user experience is enhanced.

[0073] Correspondingly, for certain target multimedia processing processes, such as some non-essential multimedia processing processes, such as special effects such as beauty, noise reduction, super-resolution and other processing processes, when using the service quality prediction model to predict the service quality indicator prediction value of the target multimedia processing process when applying the candidate multimedia processing parameters, it is also possible to predict the service quality indicator prediction value of the target multimedia processing process when adopting the determined target algorithm. No further details will be given here.

[0074] On the basis of any of the above embodiments, when global parameter configuration is performed in advance for real-time communication of terminal devices under different real-time performance parameters, the performance scores corresponding to different real-time performance parameters can be determined, and the mapping relationship between different preset performance scores and different preset multimedia processing parameter sets can be configured. When determining the corresponding preset multimedia processing parameter set in real-time communication based on the real-time performance parameters of the terminal device in real-time communication, the current performance score of the terminal device can be determined based on the real-time performance parameters, and based on the mapping relationship between the preset performance score and the preset multimedia processing parameter set, the preset multimedia processing parameter set corresponding to the current performance score can be determined, thereby realizing the selection of the preset multimedia processing parameter set.

[0075] Based on any of the above embodiments, when the initial multimedia processing parameters are applied to the target multimedia processing process in S204, if the target multimedia processing process belongs to the downlink stage of real-time communication, such as decoding, rendering, etc., its initial multimedia processing parameters may include resolution and / or frame rate, and these target multimedia processing processes cannot control the resolution and / or frame rate of the received multimedia data, but need to obtain multimedia data that meets the initial multimedia processing parameters, i.e., the resolution and / or frame rate, from the server. In other words, the multimedia data to be processed is obtained from the server according to the initial multimedia processing parameters, and the multimedia data to be processed can be obtained from the server by subscription.

[0076] Corresponding to the multimedia processing method for real-time communication described in the preceding embodiments, FIG4 is a block diagram of a multimedia processing device for real-time communication provided in an embodiment of the present disclosure. For ease of illustration, only the portions relevant to the embodiment of the present disclosure are shown. Referring to FIG4 , the multimedia processing device 400 for real-time communication includes: a collection unit 401, an initial parameter determination unit 402, a parameter adjustment unit 403, a prediction unit 404, and an execution unit 405.

[0077] The acquisition unit 401 is used to obtain real-time performance parameters of the terminal device in real-time communication;

[0078] an initial parameter determining unit 402, configured to determine initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters, and apply the initial multimedia processing parameters to the target multimedia processing process;

[0079] a parameter adjustment unit 403 configured to obtain a real-time load condition of the target multimedia processing process; and if the real-time load condition exceeds a preset load threshold, adjusting the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters;

[0080] A prediction unit 404 is configured to use a quality of service prediction model to predict a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device;

[0081] The execution unit 405 is configured to apply the candidate multimedia processing parameters to the target multimedia processing process if it is determined that the predicted value of the service quality indicator meets the preset service quality requirement.

[0082] In one or more embodiments of the present disclosure, when determining that the real-time load condition exceeds a preset load threshold, the parameter adjustment unit 403 adjusts the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters, and is configured to:

[0083] If the real-time load condition is greater than a preset load threshold upper limit, determining whether the real-time service quality indicator of the target multimedia processing process meets a preset service quality requirement; if the preset service quality requirement is not met, adjusting the initial multimedia processing parameters to reduce the real-time load condition of the target multimedia processing process, thereby obtaining the candidate multimedia processing parameters; or

[0084] If the real-time load situation is less than the preset load threshold lower limit, it is determined whether the real-time service quality indicator of the target multimedia processing process continues to meet the preset service quality requirements for a preset period of time; if it continues to meet the preset service quality requirements for a preset period of time, the initial multimedia processing parameters are adjusted to improve the real-time load situation of the target multimedia processing process, and the candidate multimedia processing parameters are obtained.

[0085] In one or more embodiments of the present disclosure, the parameter adjustment unit 403 is further configured to:

[0086] If it is determined that the service quality indicator prediction value does not meet the preset service quality requirements, the candidate multimedia processing parameters are adjusted to obtain new candidate multimedia processing parameters, and the prediction unit 404 re-predicts the service quality indicator prediction value of the target multimedia processing process when applying the new candidate multimedia processing parameters based on the new candidate multimedia processing parameters, the attribute information of the target multimedia processing process and the real-time load of the target multimedia processing process using a service quality prediction model.

[0087] In one or more embodiments of the present disclosure, the prediction unit 404, when using a service quality prediction model to predict a predicted value of a service quality indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device, is configured to:

[0088] Dividing the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process into text type input data and numerical type input data;

[0089] The text type input data and the numerical type input data are input into the service quality prediction model, and features are extracted from the text type input data and the numerical type input data respectively. The features are learned using a multi-head self-attention layer and a fully connected layer, and the service quality indicator prediction value is output through the output layer as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied.

[0090] In one or more embodiments of the present disclosure, when performing feature extraction on the text type input data and the numerical type input data respectively, the prediction unit 404 is configured to:

[0091] Vectorizing the text-type input data using the embedding layer in the quality of service prediction model, and then performing linear transformation using a linear layer to obtain features corresponding to the text-type input data; and / or

[0092] The linear layer in the service quality prediction model is used to perform linear transformation on the numerical type input data to obtain features corresponding to the numerical type input data.

[0093] In one or more embodiments of the present disclosure, when determining initial multimedia processing parameters of at least one target multimedia processing process in real-time communication based on the real-time performance parameters and applying the initial multimedia processing parameters to the target multimedia processing process, the initial parameter determination unit 402 is configured to:

[0094] determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, wherein the set of preset multimedia processing parameters includes multimedia processing parameter thresholds of a plurality of multimedia processing processes in real-time communication, wherein the plurality of multimedia processing processes includes the target multimedia processing process;

[0095] The multimedia processing parameter threshold of each multimedia processing process in the preset multimedia processing parameter set is used as an initial multimedia processing parameter and applied to the corresponding multimedia processing process.

[0096] In one or more embodiments of the present disclosure, when determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, the initial parameter determination unit 402 is configured to:

[0097] The current performance score of the terminal device is determined according to the real-time performance parameter, and the preset multimedia processing parameter set corresponding to the current performance score is determined according to a mapping relationship between the preset performance score and the preset multimedia processing parameter set.

[0098] In one or more embodiments of the present disclosure, the initial parameter determination unit 402 is further configured to:

[0099] determining whether any target multimedia processing process is started according to the real-time performance parameter; or

[0100] A target algorithm is selected from a set of candidate algorithms for any target multimedia processing process according to the real-time performance parameter, wherein the set of candidate algorithms includes candidate algorithms of different complexities.

[0101] In one or more embodiments of the present disclosure, when applying the initial multimedia processing parameters to the target multimedia processing process, the execution unit 405 is configured to:

[0102] If the target multimedia processing process belongs to the downlink phase of real-time communication, the multimedia data to be processed is obtained from the server according to the initial multimedia processing parameters.

[0103] The device provided in this embodiment can be used to execute the technical solution of the above method embodiment. Its implementation principle and technical effects are similar and will not be described in detail in this embodiment.

[0104] Referring to FIG5 , there is shown a schematic diagram of the structure of an electronic device 500 suitable for implementing an embodiment of the present disclosure. The electronic device 500 may be a terminal device or a server. The terminal device may include, but is not limited to, mobile terminals such as mobile phones, laptop computers, digital broadcast receivers, personal digital assistants (PDAs), tablet computers (Portable Android Devices, PADs), portable multimedia players (PMPs), vehicle-mounted terminals (e.g., vehicle-mounted navigation terminals), and fixed terminals such as digital TVs and desktop computers. The electronic device shown in FIG5 is merely an example and should not limit the functionality and scope of use of the embodiments of the present disclosure.

[0105] As shown in FIG5 , the electronic device 500 may include a processing device (e.g., a central processing unit, a graphics processing unit, etc.) 501, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 502 or a program loaded from a storage device 508 into a random access memory (RAM) 503. Various programs and data required for the operation of the electronic device 500 are also stored in the RAM 503. The processing device 501, the ROM 502, and the RAM 503 are connected to each other via a bus 504. An input / output (I / O) interface 505 is also connected to the bus 504.

[0106] Typically, the following devices may be connected to the I / O interface 505: an input device 506 including, for example, a touch screen, a touchpad, a keyboard, a mouse, a camera, a microphone, an accelerometer, a gyroscope, etc.; an output device 507 including, for example, a liquid crystal display (LCD), a speaker, a vibrator, etc.; a storage device 508 including, for example, a magnetic tape, a hard disk, etc.; and a communication device 509. The communication device 509 may allow the electronic device 500 to communicate with other devices wirelessly or by wire to exchange data. Although FIG5 shows an electronic device 500 having various devices, it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.

[0107] In particular, according to an embodiment of the present disclosure, the process described above with reference to the flowchart can be implemented as a computer software program. For example, an embodiment of the present disclosure includes a computer program product, which includes a computer program carried on a computer-readable medium, and the computer program includes program code for executing the method shown in the flowchart. In such an embodiment, the computer program can be downloaded and installed from the network via the communication device 509, or installed from the storage device 508, or installed from the ROM 502. When the computer program is executed by the processing device 501, the above-mentioned functions defined in the method of the embodiment of the present disclosure are performed.

[0108] It should be noted that the computer-readable medium mentioned above in the present disclosure may be a computer-readable signal medium or a computer-readable storage medium, or any combination of the two. A computer-readable storage medium may be, for example, but not limited to, an electrical, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or component, or any combination of the above. More specific examples of computer-readable storage media may include, but are not limited to: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In the present disclosure, a computer-readable storage medium may be any tangible medium that contains or stores a program that can be used by or in conjunction with an instruction execution system, device, or component. In the present disclosure, a computer-readable signal medium may include a data signal propagated in baseband or as part of a carrier wave, which carries computer-readable program code. Such a propagated data signal may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. A computer-readable signal medium may also be any computer-readable medium other than a computer-readable storage medium that can transmit, propagate, or transport a program for use by or in conjunction with an instruction execution system, apparatus, or device. The program code contained on the computer-readable medium may be transmitted using any suitable medium, including but not limited to wires, optical cables, RF (radio frequency), etc., or any suitable combination thereof.

[0109] The computer-readable medium may be included in the electronic device, or may exist independently without being incorporated into the electronic device.

[0110] The computer-readable medium carries one or more programs. When the one or more programs are executed by the electronic device, the electronic device executes the method shown in the above embodiment.

[0111] Computer program code for performing the operations of the present disclosure may be written in one or more programming languages, or a combination thereof, including object-oriented programming languages ​​such as Java, Smalltalk, C++, and conventional procedural programming languages ​​such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a stand-alone software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or server. In cases involving a remote computer, the remote computer may be connected to the user's computer through any type of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet Service Provider).

[0112] The flowcharts and block diagrams in the accompanying drawings illustrate the possible implementation architecture, functions and operations of the systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each box in the flowchart or block diagram can represent a module, program segment, or a part of code, and the module, program segment, or a part of code contains one or more executable instructions for realizing the specified logical function. It should also be noted that in some alternative implementations, the functions marked in the box can also occur in a different order than that marked in the accompanying drawings. For example, two boxes represented in succession can actually be executed substantially in parallel, and they can sometimes be executed in the opposite order, depending on the functions involved. It should also be noted that each box in the block diagram and / or flowchart, and the combination of the boxes in the block diagram and / or flowchart, can be implemented with a dedicated hardware-based system that performs the specified function or operation, or can be implemented with a combination of dedicated hardware and computer instructions.

[0113] The units involved in the embodiments described in this disclosure may be implemented in software or hardware. In some cases, the name of a unit does not limit the unit itself. For example, the first acquisition unit may also be described as a "unit for acquiring at least two Internet Protocol addresses."

[0114] The functions described above herein may be performed, at least in part, by one or more hardware logic components. For example, and without limitation, exemplary types of hardware logic components that may be used include: field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chip (SOCs), complex programmable logic devices (CPLDs), and the like.

[0115] In the context of the present disclosure, a machine-readable medium can be a tangible medium that can contain or store a program for use by or in conjunction with an instruction execution system, device or equipment. A machine-readable medium can be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium can include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium can include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.

[0116] In a first aspect, according to one or more embodiments of the present disclosure, a multimedia processing method in real-time communication is provided, which is applied to a terminal device, and the method includes:

[0117] Acquiring real-time performance parameters of the terminal device in real-time communication, determining initial multimedia processing parameters of at least one target multimedia processing process in the real-time communication based on the real-time performance parameters, and applying the initial multimedia processing parameters to the target multimedia processing process;

[0118] obtaining a real-time load condition of the target multimedia processing process; if the real-time load condition exceeds a preset load threshold, adjusting the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters;

[0119] Predicting a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied using a service quality prediction model based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device;

[0120] If it is determined that the service quality indicator prediction value meets the preset service quality requirement, the candidate multimedia processing parameter is applied to the target multimedia processing process.

[0121] According to one or more embodiments of the present disclosure, if the real-time load condition exceeds a preset load threshold, the initial multimedia processing parameters are adjusted according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters, including:

[0122] If the real-time load condition is greater than a preset load threshold upper limit, determining whether the real-time service quality indicator of the target multimedia processing process meets a preset service quality requirement; if the preset service quality requirement is not met, adjusting the initial multimedia processing parameters to reduce the real-time load condition of the target multimedia processing process, thereby obtaining the candidate multimedia processing parameters; or

[0123] If the real-time load situation is less than the preset load threshold lower limit, it is determined whether the real-time service quality indicator of the target multimedia processing process continues to meet the preset service quality requirements for a preset period of time; if it continues to meet the preset service quality requirements for a preset period of time, the initial multimedia processing parameters are adjusted to improve the real-time load situation of the target multimedia processing process, and the candidate multimedia processing parameters are obtained.

[0124] According to one or more embodiments of the present disclosure, the method further includes:

[0125] If it is determined that the service quality indicator prediction value does not meet the preset service quality requirements, the candidate multimedia processing parameters are adjusted to obtain new candidate multimedia processing parameters, and the service quality prediction model is used to predict the service quality indicator prediction value of the target multimedia processing process when the new candidate multimedia processing parameters are applied based on the new candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process.

[0126] According to one or more embodiments of the present disclosure, the method of predicting a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied using a quality of service prediction model based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device includes:

[0127] Dividing the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process into text type input data and numerical type input data;

[0128] The text type input data and the numerical type input data are input into the service quality prediction model, and features are extracted from the text type input data and the numerical type input data respectively. The features are learned using a multi-head self-attention layer and a fully connected layer, and the service quality indicator prediction value is output through the output layer as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied.

[0129] According to one or more embodiments of the present disclosure, the extracting features of the text input data and the numerical input data respectively includes:

[0130] Vectorizing the text-type input data using the embedding layer in the quality of service prediction model, and then performing linear transformation using a linear layer to obtain features corresponding to the text-type input data; and / or

[0131] The linear layer in the service quality prediction model is used to perform linear transformation on the numerical type input data to obtain features corresponding to the numerical type input data.

[0132] According to one or more embodiments of the present disclosure, determining initial multimedia processing parameters of at least one target multimedia processing process in real-time communication based on the real-time performance parameters, and applying the initial multimedia processing parameters to the target multimedia processing process includes:

[0133] determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, wherein the set of preset multimedia processing parameters includes multimedia processing parameter thresholds of a plurality of multimedia processing processes in real-time communication, wherein the plurality of multimedia processing processes includes the target multimedia processing process;

[0134] The multimedia processing parameter threshold of each multimedia processing process in the preset multimedia processing parameter set is used as an initial multimedia processing parameter and applied to the corresponding multimedia processing process.

[0135] According to one or more embodiments of the present disclosure, determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter includes:

[0136] The current performance score of the terminal device is determined according to the real-time performance parameter, and the preset multimedia processing parameter set corresponding to the current performance score is determined according to a mapping relationship between the preset performance score and the preset multimedia processing parameter set.

[0137] According to one or more embodiments of the present disclosure, the method further includes:

[0138] determining whether any target multimedia processing process is started according to the real-time performance parameter; or

[0139] A target algorithm is selected from a set of candidate algorithms for any target multimedia processing process according to the real-time performance parameter, wherein the set of candidate algorithms includes candidate algorithms of different complexities.

[0140] According to one or more embodiments of the present disclosure, applying the initial multimedia processing parameters to the target multimedia processing process includes:

[0141] If the target multimedia processing process belongs to the downlink phase of real-time communication, the multimedia data to be processed is obtained from the server according to the initial multimedia processing parameters.

[0142] In a second aspect, according to one or more embodiments of the present disclosure, a multimedia processing device for real-time communication is provided, comprising:

[0143] A collection unit, used to obtain real-time performance parameters of terminal devices in real-time communication;

[0144] an initial parameter determination unit, configured to determine initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters, and apply the initial multimedia processing parameters to the target multimedia processing process;

[0145] a parameter adjustment unit, configured to obtain a real-time load condition of the target multimedia processing process; and if the real-time load condition exceeds a preset load threshold, adjust the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters;

[0146] a prediction unit, configured to use a quality of service prediction model to predict a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied, based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device;

[0147] An execution unit is configured to apply the candidate multimedia processing parameters to the target multimedia processing process if it is determined that the service quality indicator prediction value meets the preset service quality requirement.

[0148] According to one or more embodiments of the present disclosure, when the parameter adjustment unit determines that the real-time load condition exceeds a preset load threshold, it adjusts the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters, and is configured to:

[0149] If the real-time load condition is greater than a preset load threshold upper limit, determining whether the real-time service quality indicator of the target multimedia processing process meets a preset service quality requirement; if the preset service quality requirement is not met, adjusting the initial multimedia processing parameters to reduce the real-time load condition of the target multimedia processing process, thereby obtaining the candidate multimedia processing parameters; or

[0150] If the real-time load situation is less than the preset load threshold lower limit, it is determined whether the real-time service quality indicator of the target multimedia processing process continues to meet the preset service quality requirements for a preset period of time; if it continues to meet the preset service quality requirements for a preset period of time, the initial multimedia processing parameters are adjusted to improve the real-time load situation of the target multimedia processing process, and the candidate multimedia processing parameters are obtained.

[0151] According to one or more embodiments of the present disclosure, the parameter adjustment unit is further configured to:

[0152] If it is determined that the service quality indicator prediction value does not meet the preset service quality requirements, the candidate multimedia processing parameters are adjusted to obtain new candidate multimedia processing parameters, and the prediction unit re-predicts the service quality indicator prediction value of the target multimedia processing process when applying the new candidate multimedia processing parameters based on the new candidate multimedia processing parameters, the attribute information of the target multimedia processing process and the real-time load of the target multimedia processing process using a service quality prediction model.

[0153] According to one or more embodiments of the present disclosure, the prediction unit, when using a quality of service prediction model to predict a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied, is configured to:

[0154] Dividing the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process into text type input data and numerical type input data;

[0155] The text type input data and the numerical type input data are input into the service quality prediction model, and features are extracted from the text type input data and the numerical type input data respectively. The features are learned using a multi-head self-attention layer and a fully connected layer, and the service quality indicator prediction value is output through the output layer as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied.

[0156] According to one or more embodiments of the present disclosure, when the prediction unit performs feature extraction on the text type input data and the numerical type input data respectively, it is configured to:

[0157] Vectorizing the text-type input data using the embedding layer in the quality of service prediction model, and then performing linear transformation using a linear layer to obtain features corresponding to the text-type input data; and / or

[0158] The linear layer in the service quality prediction model is used to perform linear transformation on the numerical type input data to obtain features corresponding to the numerical type input data.

[0159] According to one or more embodiments of the present disclosure, when the initial parameter determination unit determines the initial multimedia processing parameters of at least one target multimedia processing process in real-time communication based on the real-time performance parameters and applies the initial multimedia processing parameters to the target multimedia processing process, it is configured to:

[0160] determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, wherein the set of preset multimedia processing parameters includes multimedia processing parameter thresholds of a plurality of multimedia processing processes in real-time communication, wherein the plurality of multimedia processing processes includes the target multimedia processing process;

[0161] The multimedia processing parameter threshold of each multimedia processing process in the preset multimedia processing parameter set is used as an initial multimedia processing parameter and applied to the corresponding multimedia processing process.

[0162] According to one or more embodiments of the present disclosure, when determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, the initial parameter determination unit is configured to:

[0163] The current performance score of the terminal device is determined according to the real-time performance parameter, and the preset multimedia processing parameter set corresponding to the current performance score is determined according to a mapping relationship between the preset performance score and the preset multimedia processing parameter set.

[0164] According to one or more embodiments of the present disclosure, the initial parameter determination unit is further configured to:

[0165] determining whether any target multimedia processing process is started according to the real-time performance parameter; or

[0166] A target algorithm is selected from a set of candidate algorithms for any target multimedia processing process according to the real-time performance parameter, wherein the set of candidate algorithms includes candidate algorithms of different complexities.

[0167] According to one or more embodiments of the present disclosure, when applying the initial multimedia processing parameters to the target multimedia processing process, the execution unit is configured to:

[0168] If the target multimedia processing process belongs to the downlink phase of real-time communication, the multimedia data to be processed is obtained from the server according to the initial multimedia processing parameters.

[0169] In a third aspect, according to one or more embodiments of the present disclosure, there is provided an electronic device, comprising: at least one processor and a memory;

[0170] The memory stores computer-executable instructions;

[0171] The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect.

[0172] In a fourth aspect, according to one or more embodiments of the present disclosure, a computer-readable storage medium is provided, wherein the computer-readable storage medium stores computer execution instructions. When the processor executes the computer execution instructions, the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect is implemented.

[0173] In a fifth aspect, according to one or more embodiments of the present disclosure, a computer program product is provided, comprising computer execution instructions. When a processor executes the computer execution instructions, the multimedia processing method in real-time communication as described in the first aspect and various possible designs of the first aspect is implemented.

[0174] The above description is merely a preferred embodiment of the present disclosure and an illustration of the technical principles employed. Those skilled in the art should understand that the scope of disclosure involved in the present disclosure is not limited to the technical solutions formed by the specific combination of the above-mentioned technical features, but also includes other technical solutions formed by any combination of the above-mentioned technical features or their equivalents without departing from the above-mentioned disclosed concepts. For example, a technical solution formed by replacing the above-mentioned features with (but not limited to) technical features with similar functions disclosed in this disclosure.

[0175] In addition, although each operation is described in a specific order, this should not be understood as requiring these operations to be performed in the specific order shown or in a sequential order. Under certain circumstances, multitasking and parallel processing may be advantageous. Similarly, although some specific implementation details have been included in the above discussion, these should not be interpreted as limiting the scope of the present disclosure. Some features described in the context of a separate embodiment can also be implemented in a single embodiment in combination. On the contrary, the various features described in the context of a single embodiment can also be implemented in multiple embodiments individually or in any suitable sub-combination mode.

[0176] Although the subject matter has been described in language specific to structural features and / or methodological logical acts, it should be understood that the subject matter defined in the appended claims is not necessarily limited to the specific features or acts described above. Rather, the specific features and acts described above are merely example forms of implementing the claims.

Claims

1. A multimedia processing method in real-time communication, applied to a terminal device, comprising: Acquiring real-time performance parameters of the terminal device in real-time communication, determining initial multimedia processing parameters of at least one target multimedia processing process in the real-time communication based on the real-time performance parameters, and applying the initial multimedia processing parameters to the target multimedia processing process; obtaining a real-time load condition of the target multimedia processing process; if the real-time load condition exceeds a preset load threshold, adjusting the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters; Predicting a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied using a quality of service prediction model based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device; If it is determined that the service quality indicator prediction value meets the preset service quality requirement, the candidate multimedia processing parameter is applied to the target multimedia processing process.

2. The method according to claim 1, wherein If the real-time load condition exceeds a preset load threshold, the initial multimedia processing parameters are adjusted according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters, including: If the real-time load condition is greater than a preset load threshold upper limit, determining whether the real-time service quality indicator of the target multimedia processing process meets a preset service quality requirement; if the preset service quality requirement is not met, adjusting the initial multimedia processing parameters to reduce the real-time load condition of the target multimedia processing process, thereby obtaining the candidate multimedia processing parameters; or If the real-time load situation is less than the preset load threshold lower limit, it is determined whether the real-time service quality indicator of the target multimedia processing process continues to meet the preset service quality requirements for a preset period of time; if it continues to meet the preset service quality requirements for a preset period of time, the initial multimedia processing parameters are adjusted to improve the real-time load situation of the target multimedia processing process, and the candidate multimedia processing parameters are obtained.

3. The method according to claim 1 or 2, further comprising: If it is determined that the service quality indicator prediction value does not meet the preset service quality requirements, the candidate multimedia processing parameters are adjusted to obtain new candidate multimedia processing parameters, and the service quality prediction model is used to predict the service quality indicator prediction value of the target multimedia processing process when the new candidate multimedia processing parameters are applied based on the new candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process.

4. The method according to any one of claims 1 to 3, wherein: The method of predicting a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied using a service quality prediction model based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device includes: Dividing the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time load of the target multimedia processing process into text type input data and numerical type input data; The text type input data and the numerical type input data are input into the service quality prediction model, and features are extracted from the text type input data and the numerical type input data respectively. The features are learned using a multi-head self-attention layer and a fully connected layer, and the service quality indicator prediction value is output through the output layer as the service quality indicator prediction value of the target multimedia processing process when the candidate multimedia processing parameters are applied.

5. The method according to claim 4, wherein The feature extraction of the text type input data and the numerical type input data respectively includes: Vectorizing the text-type input data using the embedding layer in the quality of service prediction model, and then performing linear transformation using a linear layer to obtain features corresponding to the text-type input data; and / or The linear layer in the service quality prediction model is used to perform linear transformation on the numerical type input data to obtain features corresponding to the numerical type input data.

6. The method according to any one of claims 1 to 5, wherein: The determining, based on the real-time performance parameters, initial multimedia processing parameters of at least one target multimedia processing process in the real-time communication, and applying the initial multimedia processing parameters to the target multimedia processing process, comprises: determining a corresponding set of preset multimedia processing parameters in real-time communication according to the real-time performance parameter, wherein the set of preset multimedia processing parameters includes multimedia processing parameter thresholds of a plurality of multimedia processing processes in real-time communication, wherein the plurality of multimedia processing processes includes the target multimedia processing process; The multimedia processing parameter threshold of each multimedia processing process in the preset multimedia processing parameter set is used as an initial multimedia processing parameter and applied to the corresponding multimedia processing process.

7. The method according to claim 6, wherein: The determining, according to the real-time performance parameter, a corresponding set of preset multimedia processing parameters in real-time communication includes: The current performance score of the terminal device is determined according to the real-time performance parameter, and the preset multimedia processing parameter set corresponding to the current performance score is determined according to a mapping relationship between the preset performance score and the preset multimedia processing parameter set.

8. The method according to claim 6 or 7, further comprising: determining whether any target multimedia processing process is started according to the real-time performance parameter; or A target algorithm is selected from a set of candidate algorithms for any target multimedia processing process according to the real-time performance parameter, wherein the set of candidate algorithms includes candidate algorithms of different complexities.

9. The method according to any one of claims 1 to 8, wherein: The applying the initial multimedia processing parameters to the target multimedia processing process includes: If the target multimedia processing process belongs to the downlink phase of real-time communication, the multimedia data to be processed is obtained from the server according to the initial multimedia processing parameters.

10. A multimedia processing device for real-time communication, comprising: A collection unit configured to obtain real-time performance parameters of the terminal device in real-time communication; an initial parameter determining unit configured to determine initial multimedia processing parameters of at least one target multimedia processing process in real-time communication according to the real-time performance parameters, and apply the initial multimedia processing parameters to the target multimedia processing process; a parameter adjustment unit configured to obtain a real-time load condition of the target multimedia processing process; if the real-time load condition exceeds a preset load threshold, adjust the initial multimedia processing parameters according to the real-time service quality indicator of the target multimedia processing process to obtain candidate multimedia processing parameters; a prediction unit configured to, based on the candidate multimedia processing parameters, the attribute information of the target multimedia processing process, and the real-time performance parameters of the terminal device, use a quality of service prediction model to predict a predicted value of a quality of service indicator of the target multimedia processing process when the candidate multimedia processing parameters are applied; The execution unit is configured to apply the candidate multimedia processing parameter to the target multimedia processing process if it is determined that the service quality indicator prediction value meets the preset service quality requirement.

11. An electronic device comprising: at least one processor and memory, wherein The memory stores computer-executable instructions; The at least one processor executes the computer-executable instructions stored in the memory, so that the at least one processor performs the method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing computer-executable instructions, wherein: When a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

13. A computer program product comprising computer-executable instructions, wherein: When a processor executes the computer-executable instructions, the method according to any one of claims 1 to 9 is implemented.

Images