Media processing method and device, network equipment, storage medium and program product
By dynamically adjusting the buffer duration of audio and video streams, the problem that fixed buffer durations cannot adapt to different transmission scenarios is solved, and the transmission latency of audio and video streams is reduced.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- CHINA MOBILE COMM LTD RES INST
- Filing Date
- 2024-11-20
- Publication Date
- 2026-05-22
Smart Images
Figure CN122073618A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of video network technology, and in particular to a media processing method, apparatus, network device, storage medium and program product. Background Technology
[0002] In video networking services, audio and video streams pass through multiple processing nodes before being transmitted to the Data Channel Media Function (DCMF) network element for processing. Each processing node incorporates a jitter buffer to prevent audio and video stuttering, taking into account network jitter resistance.
[0003] Currently, during audio and video calls, in the VoNR+ media plane, the RTC video network service that connects internal and external communication, the jitter buffer size is usually set to around 150ms, which means that at least an additional 150ms of latency will be generated in the corresponding processing stage.
[0004] For highly interactive scenarios involving audio and video calls, although audio and video media processing occurs only on a single media node, the additional 150ms processing latency on that single node is unacceptable for such scenarios. For scenarios involving the integration of video services, multiple media nodes are required for serial processing. The jitter buffer time of approximately 150ms, which is based on empirical values, is also added to each media node, resulting in a significant increase in latency and causing substantial latency overhead.
[0005] Therefore, it is necessary to improve the jitter buffer duration of audio and video media service processing nodes to adapt to different audio and video transmission scenarios and improve the problem of large audio and video transmission latency. Summary of the Invention
[0006] The purpose of this application is to provide a media processing method, apparatus, network device, storage medium, and program product to solve the problem that in the processing nodes of existing audio and video streams, the jitter buffer duration is usually set to a fixed value, which cannot adapt to different audio and video transmission scenarios, resulting in a large transmission overhead for audio and video streams.
[0007] One embodiment of this application provides a media processing method, wherein the method is applied to a processing node and includes:
[0008] At each first preset time interval, the buffer duration parameter of media service transmission is obtained; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base;
[0009] The buffer duration for media service transmission is determined based on the buffer duration parameter.
[0010] Optionally, in the media processing method, the buffer duration parameter includes a media parameter latency base, and obtaining the media parameter latency base of the media service includes:
[0011] Obtain the minimum duration of each media service transmission frame and the encoding / decoding latency of each media service transmission frame;
[0012] The media parameter delay base is determined to be the sum of the minimum duration and the encoding / decoding delay.
[0013] Optionally, the media processing method further includes:
[0014] The minimum duration is determined based on the frame rate of the media service transmission.
[0015] Optionally, in the media processing method, the buffer duration parameter includes a media processing latency base, and obtaining the media processing latency base of the media service includes:
[0016] Obtain the average latency of the target algorithm executing the media service on the processing node within the second preset time period, and / or obtain the preset value of the processing latency of the target algorithm provided by the capability provider of the target algorithm;
[0017] The media processing delay baseline is determined based on the average latency or the preset processing latency value.
[0018] Optionally, in the media processing method, determining the media processing delay baseline based on the average latency or the preset processing latency value includes:
[0019] If the average latency can be obtained within a preset time range, then the media processing latency base is determined based on the average latency;
[0020] If the average latency cannot be obtained within the preset time range, the media processing latency base value is determined according to the preset processing latency value.
[0021] Optionally, in the media processing method, the buffer duration parameter includes a network condition latency base, and obtaining the network condition latency base for the media service includes:
[0022] The average transmission delay of network probe data packets is obtained when the media service is transmitted between the processing node and another processing node within a third preset time period.
[0023] The average transmission delay is set as the network condition delay base.
[0024] Optionally, in the media processing method, the target algorithm includes one or more of the following:
[0025] Rendering algorithm;
[0026] AI inference algorithms;
[0027] AI generation algorithm.
[0028] Optionally, in the media processing method, determining the buffer duration for media service transmission based on the buffer duration parameter includes:
[0029] The buffer duration for the media service transmission is determined to be equal to the sum of all parameters in the buffer duration parameter.
[0030] One embodiment of this application also provides a media processing apparatus, wherein the processing node includes:
[0031] The parameter acquisition module is used to acquire the buffer duration parameter of media service transmission at each first preset time interval; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base.
[0032] The determination module is used to determine the buffer duration of the media service transmission based on the buffer duration parameter.
[0033] One embodiment of this application also provides a network device, comprising: a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the steps of the method described in any of the preceding claims.
[0034] One embodiment of this application also provides a computer-readable storage medium, wherein a computer program is stored on the computer-readable storage medium, and the computer program, when executed by a processor, implements the steps of the method described in any of the preceding claims.
[0035] One embodiment of this application also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the method described in any one of the above.
[0036] At least one of the above-mentioned technical solutions of this application has the following beneficial effects:
[0037] The media processing method described in this application embodiment can dynamically adjust the buffer duration of media service transmission based on one or more of the service-aware parameters, namely, media parameter delay base, media processing delay base, and network condition delay base, at first preset intervals. This allows the determined buffer duration to be dynamically determined according to different audio and video transmission states, avoiding the problem that the jitter buffer duration is usually set to a fixed value, which cannot adapt to different audio and video transmission scenarios and results in large transmission overhead for audio and video streams. Attached Figure Description
[0038] Figure 1 This is a schematic diagram of one system architecture for using the media processing method described in the embodiments of this application;
[0039] Figure 2 This is a schematic flowchart of the media processing method described in the embodiments of this application;
[0040] Figure 3 This is a schematic diagram of the structure of the media processing device described in the embodiments of this application. Detailed Implementation
[0041] To make the technical problems, technical solutions and advantages of this application clearer, a detailed description will be provided below in conjunction with the accompanying drawings and specific embodiments.
[0042] The terms "first," "second," etc., used in the specification and claims of this invention are used to distinguish similar objects and not to describe a specific order or sequence. It should be understood that such terms can be used interchangeably where appropriate so that embodiments of this application can be implemented in orders other than those illustrated or described herein, and the objects distinguished by "first" and "second" are generally of the same class, not limited in number; for example, the first object can be one or more. Furthermore, in the specification and claims, "and / or" indicates at least one of the connected objects, and the character " / " generally indicates that the preceding and following objects are in an "or" relationship.
[0043] To address the problem that existing audio and video stream processing nodes typically have fixed jitter buffer durations, which cannot adapt to different audio and video transmission scenarios and result in high transmission overhead, this application provides a media processing method that can dynamically adjust the buffer duration of media service transmission based on one or more service-aware parameters, including media parameter latency base, media processing latency base, and network condition latency base. This avoids the problem of high transmission overhead caused by fixed jitter buffer durations failing to adapt to different audio and video transmission scenarios.
[0044] like Figure 1The diagram shows one system architecture of the media processing method described in this application. This system architecture can be configured as an Internet Protocol (IP) Multimedia Subsystem (IMS), a network architecture for providing multimedia communication services. In the IMS network architecture, the data channel application (DC App) can provide users with richer communication services beyond voice or video calls.
[0045] The IMS network includes one or more of the following network elements: call session control function (CSCF), IMS access media gateway (AGW), data channel signaling function (DCSF), data channel media function (DCMF), home subscriber server (HSS), IMS application server (AS), data channel application server (DCAS), and network exposure function (NEF).
[0046] The media processing method described in this application embodiment can be applied to any processing node or transmission node in the above network architecture for audio and video transmission.
[0047] It should be noted that, Figure 1 The system architecture shown is merely an example of the network architecture used in the media processing method described in this embodiment of the invention. The network architecture applicable to this application embodiment is not limited to this; any network architecture capable of realizing general video service transmission is applicable to this application embodiment. For example, the system architecture of the core network VoNR+ media plane and / or video network media processing services in a call link can also be applied to this application embodiment.
[0048] Therefore, the network architecture and business scenarios described in the embodiments of this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided in the embodiments of this application. As those skilled in the art will know, with the evolution of network architecture and the emergence of new business scenarios, the technical solutions provided in the embodiments of this application are also applicable to similar technical problems.
[0049] Figure 2 This is a flowchart illustrating the media processing method described in one embodiment, applied to a processing node, including:
[0050] S210, at each first preset time interval, obtain the buffer duration parameter of media service transmission; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base;
[0051] S220, determine the buffer duration for the media service transmission based on the buffer duration parameter.
[0052] In this application embodiment, the media services mentioned include audio and video services.
[0053] Using the media processing method described in the embodiments of this application, at each first preset time interval, the buffer duration parameter of media service transmission is obtained, and based on the obtained buffer duration parameter, the buffer duration of media service transmission is dynamically adjusted so that the determined buffer duration can be determined according to different audio and video transmission states, thereby avoiding the problem of excessive transmission delay.
[0054] In one embodiment, optionally, the buffer duration parameter includes a media parameter latency base. In step S210, obtaining the media parameter latency base for the media service includes:
[0055] Obtain the minimum duration of each media service transmission frame and the encoding / decoding latency of each media service transmission frame;
[0056] The media parameter delay base is determined to be the sum of the minimum duration and the encoding / decoding delay.
[0057] The media parameter latency base is a latency parameter that is as small as possible without affecting audio and video decoding and playback.
[0058] Optionally, the method further includes:
[0059] The minimum duration is determined based on the frame rate of the media service transmission.
[0060] In one implementation, the minimum duration is equal to 1 second / the frame rate of the media service transmission. That is, the media parameter delay base can be determined as follows: Media parameter delay base = 1 second / the frame rate of the media service transmission + the encoding / decoding delay per frame of media service transmission.
[0061] In some embodiments, the frame rate of media service transmission is optionally determined through negotiation between the core network and the terminal. Optionally, this frame rate can be obtained through Session Description Protocol (SDP) information. Wherein, during audio / video parameter switching or renegotiation, the frame rate of media service transmission will be adjusted in a timely manner. The media processing method described in this application obtains the frame rate of media service transmission at first preset intervals, and determines a buffer duration based on the frame rate, so that the determined buffer duration can adapt to parameter changes after audio / video parameter switching or renegotiation.
[0062] Optionally, the encoding and decoding latency of each media service transmission frame is determined by the terminal-side performance. Optionally, the mobile phone hardware image signal processing (ISP) performance of the corresponding playback terminal can be estimated based on the encoding and decoding compatibility list on the terminal side, and the encoding and decoding latency of each media service transmission frame can be determined based on the ISP performance.
[0063] In this implementation method, the collected media parameters are obtained at first preset intervals, and the buffer duration for media service transmission is determined based on the dynamically changing media parameters. This allows the determined buffer duration to be dynamically adjusted in real time according to the media parameters to adapt to the dynamic changes in the media parameters.
[0064] In another embodiment, the buffer duration parameter includes a media processing latency base. In step S210, obtaining the media processing latency base of the media service includes:
[0065] Obtain the average latency of the target algorithm executing the media service on the processing node within the second preset time period, and / or obtain the preset value of the processing latency of the target algorithm provided by the capability provider of the target algorithm;
[0066] The media processing delay baseline is determined based on the average latency or the preset processing latency value.
[0067] In this embodiment of the application, the media processing latency base is the minimum latency parameter generated by the processing node executing the target algorithm using the method described in this embodiment of the application.
[0068] In one implementation, optionally, the media processing latency baseline is the average latency calculated by applying the target algorithm to the processing node over a preset number of recent media service executions. The duration of these preset number of executions is the second preset duration. In this implementation, the media processing latency baseline is determined to be equal to the calculated average latency.
[0069] In another implementation, optionally, the media processing latency base is determined to be equal to the preset processing latency value of the target algorithm provided by the capability provider of the target algorithm.
[0070] The processing delay preset value can be a preset value written in the target algorithm.
[0071] In one embodiment, optionally, the media processing delay baseline is determined based on the average latency or the preset processing latency value, including:
[0072] If the average latency can be obtained within a preset time range, then the media processing latency base is determined based on the average latency;
[0073] If the average latency cannot be obtained within the preset time range, the media processing latency base value is determined according to the preset processing latency value.
[0074] Optionally, the target algorithm includes one or more of the following:
[0075] Rendering algorithm;
[0076] AI inference algorithms;
[0077] AI generation algorithm.
[0078] In this implementation method, the average latency of the target algorithm when executing media services is obtained at first preset intervals. The buffer duration of media service transmission is determined based on the average latency, so that the determined buffer duration can be dynamically adjusted in real time according to the media processing situation to adapt to the media processing status.
[0079] Optionally, if the average latency cannot be obtained within the preset time range, the average latency is determined according to the preset processing latency value of the target algorithm provided by the target algorithm capability provider, so as to ensure the correct execution of dynamically adjusting the buffer duration.
[0080] In one embodiment, optionally, the buffer duration parameter includes a network condition latency baseline. In step S210, obtaining the network condition latency baseline for the media service includes:
[0081] The average transmission delay of network probe data packets is obtained when the media service is transmitted between the processing node and another processing node within a third preset time period.
[0082] The average transmission delay is set as the network condition delay base.
[0083] In this implementation, the network condition latency baseline is calculated using network probe jitter latency. The network probe data packet is the data packet used for latency monitoring between this processing node and another processing node.
[0084] In this implementation method, the average transmission delay generated during media service transmission is obtained by intervening at a first preset time interval, and the buffer time for media service transmission is determined based on the average transmission delay, so that the determined buffer time can be dynamically adjusted in real time according to the network transmission situation to adapt to the dynamic changes in media service transmission.
[0085] In one embodiment, optionally, in step S220, determining the buffer duration for the media service transmission based on the buffer duration parameter includes:
[0086] The buffer duration for the media service transmission is determined to be equal to the sum of all parameters in the buffer duration parameter.
[0087] For example, if the buffer duration parameter includes the media parameter delay base, the media processing delay base, and the network condition delay base, then the buffer duration for the media service transmission is equal to the sum of the media parameter delay base, the media processing delay base, and the network condition delay base determined in the above manner.
[0088] In one embodiment, optionally, the method further includes:
[0089] Based on the obtained buffer duration parameters, the first preset duration is dynamically adjusted so that the adjustment cycle of the buffer duration can be dynamically adjusted according to the collected media parameters, media service processing, media service transmission, and other aspects.
[0090] Using the method described in the embodiments of this application, a jitter buffer calculation strategy is provided to support flexible setting of the jitter buffer duration. Under good network conditions, the buffer duration can be set to a minimum of 1ms, minimizing the additional latency caused by buffering. For the case of multiple nodes running serially, the latency overhead is also greatly reduced.
[0091] One embodiment of this application also provides a media processing device applied to a processing node, such as... Figure 3 As shown, the media processing device includes:
[0092] The parameter acquisition module 310 is used to acquire the buffer duration parameter of media service transmission at intervals of a first preset duration; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base.
[0093] The determining module 320 is used to determine the buffer duration of the media service transmission based on the buffer duration parameter.
[0094] Using the device described in this embodiment, the buffer duration parameter of media service transmission is obtained at each first preset time interval. Based on the obtained buffer duration parameter, the buffer duration of media service transmission is dynamically adjusted so that the determined buffer duration can be determined according to different audio and video transmission states, thereby avoiding the problem of excessive transmission delay.
[0095] Optionally, in the media processing apparatus, the buffer duration parameter includes a media parameter latency base, and the parameter acquisition module 310 acquires the media parameter latency base of the media service, including:
[0096] Obtain the minimum duration of each media service transmission frame and the encoding / decoding latency of each media service transmission frame;
[0097] The media parameter delay base is determined to be the sum of the minimum duration and the encoding / decoding delay.
[0098] Optionally, in the media processing apparatus, the parameter acquisition module 310 is further configured to:
[0099] The minimum duration is determined based on the frame rate of the media service transmission.
[0100] Optionally, in the media processing apparatus, the buffer duration parameter includes a media processing latency base, and the parameter acquisition module 310 acquires the media processing latency base of the media service, including:
[0101] Obtain the average latency of the target algorithm executing the media service on the processing node within the second preset time period, and / or obtain the preset value of the processing latency of the target algorithm provided by the capability provider of the target algorithm;
[0102] The media processing delay baseline is determined based on the average latency or the preset processing latency value.
[0103] Optionally, in the media processing apparatus, the parameter acquisition module 310 determines the media processing delay baseline based on the average delay or the preset processing delay value, including:
[0104] If the average latency can be obtained within a preset time range, then the media processing latency base is determined based on the average latency;
[0105] If the average latency cannot be obtained within the preset time range, the media processing latency base value is determined according to the preset processing latency value.
[0106] Optionally, in the media processing apparatus, the buffer duration parameter includes a network condition latency baseline, and the parameter acquisition module 310 acquires the network condition latency baseline of the media service, including:
[0107] The average transmission delay of network probe data packets is obtained when the media service is transmitted between the processing node and another processing node within a third preset time period.
[0108] The average transmission delay is set as the network condition delay base.
[0109] Optionally, in the media processing apparatus, the target algorithm includes one or more of the following:
[0110] Rendering algorithm;
[0111] AI inference algorithms;
[0112] AI generation algorithm.
[0113] Optionally, in the media processing apparatus, the determining module 320 determines the buffer duration of the media service transmission based on the buffer duration parameter, including:
[0114] The buffer duration for the media service transmission is determined to be equal to the sum of all parameters in the buffer duration parameter.
[0115] One embodiment of this application also provides a network device, which includes a processor, a memory, and a program stored in the memory and executable on the processor, wherein the program, when executed by the processor, implements the media processing method as described in any of the preceding claims.
[0116] The specific implementation of the media processing method by the program running on the processor of the network device can be found in the detailed description of the media processing method when applied to the above-mentioned processing node, and will not be repeated here.
[0117] In addition, specific embodiments of this application also provide a readable storage medium having a computer program stored thereon, wherein the program, when executed by a processor, implements the steps of the media processing method as described in any of the above.
[0118] Specifically, the readable storage medium is applied to the aforementioned processing node. When applied to the processing node, the execution steps in the corresponding media processing method are described in detail above and will not be repeated here.
[0119] Another embodiment of the present invention also provides a computer program product, which includes computer instructions that, when executed by a processor, implement the steps of the media processing method as described above.
[0120] Optionally, embodiments of this application may take the form of a computer program product implemented on one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) containing computer-usable program code.
[0121] The computer program product described in this embodiment of the invention includes computer instructions that, when executed by a processor, implement the various processes of the method embodiments shown above and achieve the same technical effects. To avoid repetition, these will not be repeated here.
[0122] In the several embodiments provided in this application, it should be understood that the disclosed methods and apparatus can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between devices or units may be electrical, mechanical, or other forms.
[0123] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can be physically comprised separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or in the form of hardware plus software functional units.
[0124] The integrated units implemented as software functional units described above can be stored in a computer-readable storage medium. These software functional units, stored in a storage medium, include several instructions that cause a computer device (which may be a personal computer, server, or network device, etc.) to execute some steps of the transmission and reception methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.
[0125] The above describes the preferred embodiments of this application. It should be noted that those skilled in the art can make several improvements and modifications without departing from the principles described in this application, and these improvements and modifications should also be considered within the scope of protection of this application.
Claims
1. A media processing method, characterized in that, Applied to processing nodes, including: At each first preset time interval, the buffer duration parameter for media service transmission is obtained; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base. The buffer duration for media service transmission is determined based on the buffer duration parameter.
2. The media processing method according to claim 1, characterized in that, The buffer duration parameter includes the media parameter latency base. Obtaining the media parameter latency base for the media service includes: Obtain the minimum duration of each media service transmission frame and the encoding / decoding latency of each media service transmission frame; The media parameter delay base is determined to be the sum of the minimum duration and the encoding / decoding delay.
3. The media processing method according to claim 2, characterized in that, The method further includes: The minimum duration is determined based on the frame rate of the media service transmission.
4. The media processing method according to claim 1, characterized in that, The buffer duration parameter includes the media processing latency base, which is used to obtain the media processing latency base for the media service, including: Obtain the average latency of the target algorithm executing the media service on the processing node within the second preset time period, and / or obtain the preset value of the processing latency of the target algorithm provided by the capability provider of the target algorithm; The media processing delay baseline is determined based on the average latency or the preset processing latency value.
5. The media processing method according to claim 4, characterized in that, The media processing latency base is determined based on the average latency or the preset processing latency value, including: If the average latency can be obtained within a preset time range, then the media processing latency base is determined based on the average latency; If the average latency cannot be obtained within the preset time range, the media processing latency base value is determined according to the preset processing latency value.
6. The media processing method according to claim 1, characterized in that, The buffer duration parameter includes a network condition latency base, which is used to obtain the network condition latency base for media services, including: The average transmission delay of network probe data packets is obtained when the media service is transmitted between the processing node and another processing node within a third preset time period. The average transmission delay is set as the network condition delay base.
7. The media processing method according to claim 4 or 5, characterized in that, The target algorithm includes one or more of the following: Rendering algorithm; AI inference algorithms; AI generation algorithm.
8. The media processing method according to claim 1, characterized in that, Determining the buffer duration for media service transmission based on the buffer duration parameter includes: The buffer duration for the media service transmission is determined to be equal to the sum of all parameters in the buffer duration parameter.
9. A media processing device, characterized in that, Applied to processing nodes, including: The parameter acquisition module is used to acquire the buffer duration parameter of media service transmission at each first preset time interval; wherein, the buffer duration parameter includes one or more of the following parameters: media parameter delay base, media processing delay base, and network condition delay base. The determination module is used to determine the buffer duration of the media service transmission based on the buffer duration parameter.
10. A network device, characterized in that, include: A processor, a memory, and a program stored in the memory and executable on the processor, the program, when executed by the processor, implementing the steps of the method as described in any one of claims 1 to 8.
11. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when executed by a processor, implements the steps of the method as described in any one of claims 1 to 8.
12. A computer program product, characterized in that, Includes computer instructions that, when executed by a processor, implement the steps of the method as described in any one of claims 1 to 8.