Media content processing method and related device

Abstract

The present application discloses a media content processing method and a related device. The method may comprise: according to content attribute information of media content to be processed, allocating a memory pool for executing a media application processing task, the media application processing task comprising at least one content processing sub-task arranged according to a preset execution order; for each content frame unit among at least one content frame unit, determining, from the memory pool, a memory block to be used for processing the content frame unit; executing, for the content frame unit by means of the memory block to be used, the content processing sub-task in the media application processing task; and when the execution of the content processing sub-task is completed, on the basis of the preset execution order, multiplexing the memory block to be used to execute a new content processing sub-task in the media application processing task, so as to obtain a final content frame unit after execution of all of the content processing sub-tasks in the media application processing task is completed. The present application can reduce frequent memory allocation application requests and releases among the content processing sub-tasks, which can improve media application processing performance.

Description

Media content processing method and related equipment

[0001] This application claims priority to the Chinese patent application filed with the China Patent Office on June 5, 2023, application number 202310660301.6, and application name “Media Content Processing Method and Related Equipment”, all contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of computer technology, and in particular to media content processing technology. Background Art

[0003] With the development of computer technology and mobile communication technology, various new media platforms have emerged, and more and more users are publishing various multimedia content (such as short videos) through these new media platforms. To ensure the quality of multimedia content, various types of application processing are required, such as image quality enhancement and intelligent editing. Among them, image quality enhancement can include frame insertion, text enhancement, denoising, color enhancement, compression repair, etc. Intelligent editing can include intelligent subtitle addition, real-time live broadcast editing, watermark removal, video screenshots, etc.

[0004] The basic operations of multimedia content processing can include content file input, decapsulation, decoding, content encoding, container format packaging, content rendering, and various types of media application processing. In the current media content processing framework, these media application processes that process media content are independent and self-contained. Each media application process will re-allocate and release memory. In other words, each media application process is independent in memory allocation and each media application process will independently request and reclaim memory. This leads to frequent memory allocation and release, resulting in poor media application processing performance.

[0005] Summary of the Invention

[0006] An embodiment of the present application provides a media content processing method and related equipment. The related equipment may include a media content processing device, an electronic device, a computer-readable storage medium and a computer program product, which can reduce the frequent memory allocation application and release between various content processing subtasks, thereby improving media application processing performance.

[0007] An embodiment of the present application provides a media content processing method, which is executed by an electronic device and includes:

[0008] Acquire media content to be processed, where the media content to be processed includes at least one content frame unit;

[0009] Allocating a memory pool for executing a media application processing task based on content attribute information of the media content to be processed, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information;

[0010] For each content frame unit of the at least one content frame unit, determining a memory block to be used for processing the content frame unit from the memory pool;

[0011] executing a content processing subtask in the media application processing task on the content frame unit through the to-be-used memory block;

[0012] When the content processing subtask is completed, based on the preset execution order, the memory block to be used is reused to execute the new content processing subtask in the media application processing task on the processed content frame unit, thereby obtaining the final content frame unit after executing all the content processing subtasks in the media application processing task.

[0013] Accordingly, an embodiment of the present application provides a media content processing device, which is deployed on an electronic device and includes:

[0014] An acquisition unit, configured to acquire media content to be processed, wherein the media content to be processed includes at least one content frame unit;

[0015] an allocating unit, configured to allocate a memory pool for executing a media application processing task based on content attribute information of the media content to be processed, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information;

[0016] a memory block determining unit, configured to determine, for each content frame unit in the at least one content frame unit, a memory block to be used for processing the content frame unit from the memory pool;

[0017] A first execution unit is configured to execute a content processing subtask in the media application processing task on the content frame unit using the to-be-used memory block;

[0018] The second execution unit is used to reuse the memory block to be used to execute a new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain a final content frame unit after executing all content processing subtasks in the media application processing task.

[0019] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the media content processing method provided in the embodiment of the present application.

[0020] An embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps of the media content processing method provided in the embodiment of the present application.

[0021] In addition, an embodiment of the present application also provides a computer program product, including a computer program, which, when executed by a processor, implements the steps in the media content processing method provided in the embodiment of the present application.

[0022] The embodiment of the present application provides a media content processing method and related equipment; it can obtain the media content to be processed, and the media content to be processed includes at least one content frame unit; according to the content attribute information of the media content to be processed, a memory pool for executing the media application processing task is allocated, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit in the at least one content frame unit, a memory block to be used for processing the content frame unit is determined from the memory pool; the content processing subtask in the media application processing task is executed on the content frame unit through the memory block to be used; when the content processing subtask is completed, the memory block to be used is reused based on the preset execution order to execute the new content processing subtask in the media application processing task on the processed content frame unit, and the final content frame unit after executing all the content processing subtasks in the media application processing task is obtained. The present application can execute each content processing subtask through memory reuse, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0023] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the following briefly introduces the drawings required for use in the description of the embodiments. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0024] FIG1a is a schematic diagram of a scenario of a media content processing method provided in an embodiment of the present application;

[0025] FIG1b is a flow chart of a media content processing method provided by an embodiment of the present application;

[0026] FIG1c is a framework diagram of a media content processing method provided in an embodiment of the present application;

[0027] FIG1d is an illustration of a media content processing method provided in an embodiment of the present application;

[0028] FIG1e is another illustration of the media content processing method provided in an embodiment of the present application;

[0029] FIG1f is another framework diagram of the media content processing method provided in an embodiment of the present application;

[0030] FIG2 is another flow chart of the media content processing method provided in an embodiment of the present application;

[0031] FIG3 is a schematic structural diagram of a media content processing device provided in an embodiment of the present application;

[0032] FIG4 is a schematic structural diagram of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0033] The following will be combined with the drawings in the embodiments of this application to clearly and completely describe the technical solutions in the embodiments of this application. Obviously, the embodiments described are only part of the embodiments of this application, not all of the embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without making creative efforts are within the scope of protection of this application.

[0034] The embodiments of the present application provide a media content processing method and related equipment, which may include a media content processing device, an electronic device, a computer-readable storage medium, and a computer program product. The media content processing device may be integrated into an electronic device, which may be a terminal or a server.

[0035] It is understandable that the media content processing method of this embodiment can be executed on a terminal, on a server, or jointly by a terminal and a server. The above examples should not be construed as limiting the present application.

[0036] As shown in Figure 1a, a media content processing method is performed jointly by a terminal and a server. The media content processing system provided in the embodiment of the present application includes a terminal 10 and a server 11. The terminal 10 and the server 11 are connected via a network, such as a wired or wireless network, wherein the media content processing device can be integrated into the server.

[0037] The server 11 can be configured to: obtain media content to be processed, the media content to be processed including at least one content frame unit; allocate a memory pool for executing a media application processing task based on content attribute information of the media content to be processed, the media application processing task including at least one content processing subtask arranged in a preset execution order, and the memory pool including at least one memory block corresponding to the content attribute information; determine, for each content frame unit in the at least one content frame unit, a to-be-used memory block from the memory pool for processing the content frame unit; execute the content processing subtask in the media application processing task on the content frame unit using the to-be-used memory block; when execution of the content processing subtask is completed, reuse the to-be-used memory block to execute a new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order, thereby obtaining a final content frame unit in which all content processing subtasks in the media application processing task have been executed; and after completing the media application processing tasks for all content frame units, output the processed media content to be processed and release the to-be-used memory blocks occupied during the processing back to the memory pool, the processed media content to be processed including each final content frame unit. The server 11 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0038] The terminal 10 can be configured to receive the processed media content output by the server 11, and render and display the processed media content. The terminal 10 can include a mobile phone, an intelligent voice interaction device, a smart home appliance, an in-vehicle terminal, an aircraft, a tablet computer, a laptop computer, or a personal computer (PC). The terminal 10 can also be configured with a client, which can be an application client or a browser client.

[0039] The media content processing method provided in the embodiments of the present application relates to cloud computing in the field of cloud technology.

[0040] Cloud technology refers to a hosting technology that unifies hardware, software, and network resources within a wide area network (WAN) or local area network (LAN) to enable data computing, storage, processing, and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, and application technology based on the cloud computing business model. It can form a resource pool that can be used on demand with flexibility and convenience. Cloud computing technology will become a crucial support. Backend services for technical network systems, such as video websites, image websites, and more portals, require extensive computing and storage resources. With the rapid development and application of the internet industry, every item will likely have its own unique identification mark and will need to be transmitted to backend systems for logical processing. Data of varying levels will be processed separately, and data from all industries will require a strong system backend, which can only be achieved through cloud computing.

[0041] Cloud computing refers to the delivery and usage model of IT infrastructure, enabling on-demand, scalable access to required resources over the Internet. In a broader sense, cloud computing refers to the delivery and usage model of services, enabling on-demand, scalable access to required services over the Internet. These services can be IT-related, software-related, internet-related, or other services. Cloud computing is the product of the convergence of traditional computer and network technologies, including grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. Driven by the growth of the Internet, real-time data streams, and the diversification of connected devices, as well as the demand for search services, social networks, mobile commerce, and open collaboration, cloud computing has rapidly developed. Unlike previous parallel and distributed computing, the emergence of cloud computing will fundamentally revolutionize the entire Internet model and enterprise management model.

[0042] It should be noted that the order of description of the following embodiments is not intended to limit the preferred order of the embodiments.

[0043] This embodiment will be described from the perspective of a media content processing device. The media content processing device may be integrated into an electronic device, such as a server or a terminal.

[0044] As shown in FIG1b , the specific process of the media content processing method may be as follows:

[0045] 101. Acquire media content to be processed, where the media content to be processed includes at least one content frame unit.

[0046] Among them, the media content to be processed can be media content that needs to be processed by the method provided in the embodiment of the present application. The embodiment of the present application does not limit the type of media content to be processed. For example, it can be video type content, which can include image content and audio content. In some embodiments, the media content to be processed can also include only audio content or image content, and this embodiment does not impose specific restrictions on this. The media content to be processed can be media content that needs to perform media application processing tasks, specifically, it can be media content that has been decapsulated and decoded.

[0047] A content frame unit may include only one content frame or multiple content frames. For example, a content frame unit may include all frames in the interval (Group of Pictures, GOP) between two I frames (key frames). A GOP refers to the time interval between two I frames in video encoding.

[0048] If the media content to be processed is image content, the content frame unit may be an image frame unit; if the media content to be processed is audio content, the content frame unit may be an audio frame unit.

[0049] In a specific application scenario, as shown in Figure 1c, which is the basic architecture diagram of the media processing framework, the basic process of multimedia content processing may include content file input (Input File), decapsulation (demuxing), decoding (decoding), various types of media application processing, content encoding (encoding), container format encapsulation and packaging, and content rendering. The content files here can be audio and video files.

[0050] Among them, the input of the decapsulation module is the Input File source container format (such as mp4, mp3, ts, FLV, etc.), and the output can be Advanced Audio Coding (AAC), Opus, H.264, H.265, H.266, AV1 and other video bare streams. The output audio of the decoding module is PCM, and the video is YUV bare stream. The input and output audio of the avfilter apply effects application (application effect processing module, that is, the above-mentioned media application processing) is PCM, and the video is YUV. The input audio of the content encoding module is PCM, and the video is YUV, and the output is a coded compressed audio and video compressed bare stream such as AAC, Opus, H.264, H.265, H.266, AV1, etc. The input of the container format encapsulation and packaging module (i.e., the mixed stream encapsulation module) is the raw stream after audio and video compression such as AAC, Opus, H.264, H.265, H.266, AV1, and the output is the container encapsulated such as mp4, mp3, ts, FLV, etc.

[0051] The container format encapsulation and packaging module outputs standard container formats such as MP4, MOV, TS, and AAC. The resulting audio and video packets can be sent to a terminal. Upon receiving the packets, the terminal de-encapsulates them according to the standard container format, decodes them, and renders them for playback. This terminal can include mobile phones, computers, controllers, televisions, and virtual reality display devices, though this embodiment does not limit this.

[0052] PCM (Pulse Code Modulation) is a method for digitizing analog signals. YUV (Y stands for luminance or luma; U and V stand for chrominance or chroma) is a color encoding system primarily used in video and graphics processing pipelines.

[0053] The decapsulation module separates the input encapsulated data into compressed audio and video streams. There are many different encapsulation formats, such as MP4, MKV, RMVB, TS, FLV, and AVI. Its function is to combine the compressed video and audio data into a specific format. For example, after decapsulation, FLV data will output an H.264-encoded video stream and an AAC-encoded audio stream.

[0054] The decoding module decodes compressed video / audio data into uncompressed raw video / audio data. Through decoding, compressed video data is output as uncompressed color data, such as YUV420P, red, green, blue (RGB), etc.; compressed audio data is output as uncompressed audio sampling data, such as PCM data.

[0055] Among them, the input and output audio format of the avfilter apply effects application processing flow is PCM, and the video format is YUV. It can perform audio and video quality enhancement and intelligent editing processing based on audio and video metadata such as audio sampling frequency and video resolution. For example, image quality enhancement may include super-resolution, frame insertion, color HDR (High Dynamic Range Imaging), face enhancement, text enhancement, denoising, deburring, de-jittering, color enhancement, compression repair, review / quality inspection, and high-resistance digital watermarking; intelligent editing processing may include intelligent orchestration pipelines, intelligent subtitles, live real-time editing, splicing, abnormal container / audio and video repair, highlights, watermark removal, recording / screenshot / mixing, digital twins, etc. The main focus of this application is the avfilter apply effects application. The avfilter apply effects application can specifically include filter applications such as super-resolution, frame insertion, enhancement, color HDR, watermarking, splicing, highlights, recording screenshots, mixing, etc. (one Filter application corresponds to one content processing subtask). The processing of these Filter applications is independent, and the input and output are both audio is PCM and video is YUV. In current related technologies, each filter application is a self-closed loop, and each filter application will re-allocate and release memory during processing. Existing filter processing frameworks such as FFmpeg, Gstreamer, OpenCV, etc. are independent in memory allocation and do not reuse memory. Each application processing will independently re-apply for allocation and reclaim memory, which will lead to frequent memory allocation and reclaim. Frequent memory access allocation and release will lead to inefficient filter application processing performance. The memory performance of this type of framework architecture in high-sampling-rate and high-resolution audio and video application scenarios (such as 4K, 8K, etc.) is increasingly becoming a bottleneck for real-time application processing.

[0056] Among them, Filter is a filter that can provide many audio and video special effects processing functions, such as video scaling, interception, flipping, overlaying, etc.

[0057] Among them, compared with ordinary images, HDR can provide a wider color range and image details, improve the image light and dark contrast, greatly restore the real environment, and present extremely high image quality.

[0058] The audio sampling frequency (sampling rate) refers to how many times a recording device samples an analog signal per unit time. The higher the sampling frequency, the more realistic and natural the mechanical waveform. On today's mainstream capture cards, the sampling frequencies are generally divided into five levels: 8000Hz, 11025Hz, 22050Hz, 24000Hz, 44100Hz, and 48000Hz. 11025Hz achieves the sound quality of AM broadcasting, while 22050Hz and 24000Hz reach the sound quality of FM broadcasting. 44100Hz is the theoretical limit for CD sound quality, and 48000Hz is even more precise.

[0059] Resolution is defined as the number of pixels organized or arranged by width and height on a screen or image. Specifically, resolution measures the number of pixels in an image; the more pixels, the higher the quality. The most commonly used video resolutions are called "formats," technically known as Ultra High Definition (UHD) or Full HD. Common video resolutions include: 480p (640×480), 720p (1280×720), 1080p (1920×1080), 4K (3840×2160), and 8K (7680×4320).

[0060] This application proposes a memory pool allocation solution that reuses the memory pool when processing all filter applications after the decoding module (Decoder), reducing the engineering performance consumption caused by frequent memory allocation and release between filter applications. This can improve engineering performance by 20% or more in video filter processing application scenarios such as 4K and 8K high-resolution and high frame refresh rate (Frame per Second, FPS). The solution provided by this application has been well applied in video cloud media processing business products and has also been well applied in customers' 8K projects.

[0061] FPS stands for "frame refresh rate (measured in Hz)." In the field of graphics, FPS refers to the number of frames transmitted per second (fps), more commonly referred to as the number of frames in an animation or video. FPS measures the amount of information used to store and display dynamic video. The higher the number of frames per second, the smoother the displayed motion.

[0062] 102. Allocate a memory pool for executing a media application processing task based on content attribute information of the media content to be processed, where the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0063] The media application processing task is to perform media processing on each frame unit of the media content to be processed. Media Processing Service (MPS) can refer to the container format decapsulation, decoding, enhancement, and application processing (such as watermarking, video screenshots, smart cover, smart editing, image quality enhancement, etc.) of audio and video files, as well as audio and video codec format transcoding, packaging, and distribution processing.

[0064] The memory pool for executing the media application processing task can be applied for allocation based on the content attribute information of the media content to be processed, which may be audio sampling frequency or video resolution.

[0065] Memory pools are a type of memory allocation method, also known as fixed-size-blocks allocation. Current technologies typically use APIs (Application Programming Interfaces) such as new and malloc to request memory allocation. This approach has the disadvantage that, because the size of the allocated memory blocks varies, frequent use can lead to significant memory fragmentation. This fragmentation leads to the need for memory release, cleanup, and consolidation when insufficient memory is available. This consumes significant memory defragmentation overhead, which in turn reduces performance.

[0066] Because memory allocation and release are both inefficient, this application can initially allocate a larger block of memory (i.e., a memory pool; specifically, if this block of memory is insufficient, memory can be redistributed). Then, during the execution of tasks, the corresponding memory resources can be retrieved from this block of memory each time they are needed. Only after all media application processing tasks have been completed will the memory blocks occupied during the processing be released to the operating system. This reduces the number of memory allocation and release operations, thereby improving efficiency.

[0067] The memory pool may include at least one memory block corresponding to content attribute information. Specifically, the size of each memory block may correspond to the content attribute information of the media content to be processed (e.g., audio sampling rate, video resolution, etc.). Each memory block in the memory pool is the same size, ensuring a fixed memory size each time it is allocated, thus preventing the possibility of memory fragmentation.

[0068] In this embodiment, the media application processing task includes multiple links, each of which can be regarded as a content processing subtask. Different content processing subtasks can perform different types of content processing on content frame units. For example, the content processing subtask can be super-resolution processing, interpolation processing, or color enhancement processing.

[0069] In a possible implementation, in this embodiment, the step of “allocating a memory pool for executing the media application processing task based on content attribute information of the media content to be processed” may include:

[0070] Obtaining a preset mapping relationship set, the preset mapping relationship set including mapping relationships between preset content attributes and preset memory pools;

[0071] Querying a preset memory pool corresponding to content attribute information of the to-be-processed media content in a preset mapping relationship set;

[0072] Based on the queried preset memory pool, a memory pool is allocated for executing the media application processing task.

[0073] The preset mapping relationship set may be a relationship table between preset content attributes and preset memory pools. Based on the content attribute information, the corresponding preset memory pool may be queried from the relationship table.

[0074] The queried preset memory pool may be determined as the memory pool for executing the media application processing task.

[0075] In some embodiments, the memory resources of the memory pool can be pre-divided into at least one memory block corresponding to content attribute information (such as frame resolution size) to improve memory allocation efficiency. For example, the memory pool corresponding to the 8K video resolution can be divided based on the 8K video resolution to obtain at least one memory block corresponding to the 8K video resolution.

[0076] In a possible implementation, in this embodiment, the media content processing method may further include:

[0077] When no preset memory pool corresponding to the content attribute information is found in the preset mapping relationship set, the memory application size is determined based on the content attribute information of the to-be-processed media content and the preset processing duration of the content;

[0078] Based on the memory request size, a memory pool is allocated for executing media application processing tasks.

[0079] The preset processing content duration can be set based on actual conditions. It can also be automatically set based on the video GOP size and machine memory configuration. For example, if the GOP size is 2 seconds and the machine memory is sufficient, the preset processing content duration can be set to 2 seconds. The video GOP size is the time interval between two I-frames in the video encoding.

[0080] If the media content to be processed is image content, the content attribute information may be the image size of the image frame unit, image color sampling information, image pixel depth, and frame refresh frequency.

[0081] Image pixel bit depth refers to the bit depth required for each pixel in a computer when recording the color of a digital image. Computers are able to display color by using a unit of measurement called a "bit" to record the color data. When this data is recorded in a computer according to a specific format, it forms a computer file for a digital image. A "bit" is used to record the color value of each pixel. The richer the image's colors, the more "bits" are used. Specifically, N bits represent a single color containing 2N different shades of color. The number of bits used for each pixel in a computer is called the "bit depth."

[0082] Image color sampling information may refer to the sampling method of a YUV image, and the sampling methods may mainly include YUV 4:4:4 sampling, YUV 4:2:2 sampling, and YUV 4:2:0 sampling. Specifically, Y represents brightness (Luminance, Luma), while U and V represent chrominance (Chrominance, Chroma). Chroma defines two aspects of color: hue and saturation. For a YUV image, not every pixel needs to contain the three components Y, U, and V. Depending on the sampling method, each Y component may correspond to its own UV component, or several Y components may share the UV component. Regardless of the sampling method, the brightness is fully sampled, and the difference lies in the sampling rate of the U and V components.

[0083] Among them, the YUV 4:4:4 sampling method, each Y corresponds to a set of UV components, and the sampling ratios of the three components Y, U, and V are the same. Therefore, in the generated image, the three component information of each pixel is complete, all 8 bits, that is, one byte.

[0084] In the YUV 4:2:2 sampling method, every two Y components share a set of UV components. The UV component is half the sampled Y component, and the Y and UV components are sampled at a ratio of 2:1. If there are 10 pixels in the horizontal direction, then 10 Y components are sampled, and only 5 UV components are sampled. Specifically, it can be understood that the Y component is sampled for each pixel, while the U and V components are sampled every other pixel.

[0085] In the YUV 4:2:0 sampling method, every four Y components share a set of UV components, and the UV components are only one-fourth of the Y components. This does not mean that only the U component is sampled and the V component is not sampled. Instead, it means that when scanning each line, only one chrominance component (U or V) is scanned and sampled with the Y component in a 2:1 ratio. For example, when scanning the first line, YU is sampled in a 2:1 ratio, and when scanning the second line, YV components are sampled in a 2:1 ratio. For each chrominance component, its horizontal and vertical sampling ratio is 2:1 compared to the Y component.

[0086] Among them, if the media content to be processed is audio content, the content attribute information can be the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit. The audio sampling bit depth (bit depth) refers to the number of bits used to represent the amplitude at a certain point in time; the more bits there are, the more accurately the amplitude changes that can be represented.

[0087] In a possible implementation, in this embodiment, the media content to be processed includes image content, and the content frame unit is an image frame unit of the image content;

[0088] The step of “allocating a memory pool for executing the media application processing task based on the content attribute information of the media content to be processed” may include:

[0089] Calculate the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel depth of the image frame unit;

[0090] Determine the memory application size based on the memory capacity and the frame refresh frequency corresponding to the image frame unit;

[0091] Based on the memory request size, a memory pool is allocated for performing media application processing tasks on image content.

[0092] The image size, image color sampling information, and image pixel depth of the image frame unit can be fused to obtain the memory capacity occupied by the image frame unit; this fusion method can be multiplication. After obtaining the memory capacity occupied by the image frame unit, it can be multiplied by the frame refresh rate of the image frame unit and the preset processing content duration to obtain the memory request size, which is the size of the memory pool to be requested.

[0093] For example, the image size of an image frame unit is 7680*4320, and the image pixel bit depth is 10 bits. In the case of a 10-bit image pixel depth, two 2-byte bytes can be allocated to each pixel. In addition, the image color sampling information can be YUV 4:2:2 sampling. In YUV 4:2:2 sampling, the UV component is half the sampling of the Y component. When the number of bytes occupied by Y is 1, the number of bytes occupied by UV is 0.5. Therefore, the image color sampling information can be 1.5 bytes. The memory capacity occupied by the image frame unit can be 7680*4320*1.5*2*8 bits = 796,262,400 bits. If the preset processing content duration is 2 seconds and the frame refresh rate is 50 FPS, the memory requested size can be: 796,262,400 bits*2*50 = 79,626,240,000 bits.

[0094] In a possible implementation, in this embodiment, the media content to be processed includes audio content, and the content frame unit is an audio frame unit of the audio content;

[0095] The step of “allocating a memory pool for executing the media application processing task based on the content attribute information of the media content to be processed” may include:

[0096] Calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit;

[0097] Determine the memory size to be applied based on the memory capacity and the preset processing time.

[0098] Based on the memory request size, a memory pool is allocated for performing media application processing tasks on audio content.

[0099] Among them, the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit can be fused to calculate the memory capacity occupied by the audio frame unit. Among them, the fusion method can be multiplication. After obtaining the memory capacity occupied by the audio frame unit, it can be multiplied by the preset processing content duration to arrive at the memory application size, which is the size of the memory pool to be applied for. The preset processing content duration can be set according to actual conditions, and this embodiment does not impose any specific restrictions on this.

[0100] For example, if the audio sampling frequency corresponding to the audio frame unit is 8K and the audio sampling bit depth is 8 bits, the memory capacity occupied by the audio frame unit can be 8000*8bit=64000bit; if the preset processing content length is 2 seconds, the memory application size can be 64000bit*2=128000bit.

[0101] 103. For each content frame unit in the at least one content frame unit, determine a memory block to be used for processing the content frame unit from the memory pool.

[0102] Memory blocks in the memory pool may be marked with usage status information, including idle and occupied states. In this embodiment of the present application, a memory block to be used may be selected from the idle memory blocks. In one possible implementation, a memory block corresponding to content attribute information (such as audio sampling frequency or video resolution) of the content frame unit to be processed may also be selected.

[0103] In a possible implementation, in some embodiments, before the step of “determining a memory block to be used for processing the content frame unit from the memory pool”, the following steps may also be included:

[0104] Set the preset free memory queue;

[0105] Storing the memory addresses of the free memory blocks in the memory pool in a preset free memory queue;

[0106] The step of “determining a memory block to be used for processing the content frame unit from the memory pool” may include:

[0107] Take out the memory address to be used from the preset free memory queue;

[0108] A memory block corresponding to the memory address to be used is determined as the memory block to be used for processing the content frame unit.

[0109] 104. Execute the content processing subtask in the media application processing task on the content frame unit using the to-be-used memory block.

[0110] 105. When the content processing subtask is completed, based on the preset execution order, the memory block to be used is reused to execute a new content processing subtask in the media application processing task on the processed content frame unit, thereby obtaining a final content frame unit that has executed all content processing subtasks in the media application processing task.

[0111] In a possible implementation, in this embodiment, the step of “performing the content processing subtask in the media application processing task on the content frame unit using the to-be-used memory block” may include:

[0112] Determine a current content processing subtask from the media application processing tasks based on a preset execution order;

[0113] Execute the current content processing subtask on the content frame unit through the memory block to be used to obtain the processed content frame unit;

[0114] The step of "when the content processing subtask is completed, reusing the to-be-used memory block based on the preset execution order to execute a new content processing subtask in the media application processing task on the processed content frame unit, thereby obtaining a final content frame unit after executing all content processing subtasks in the media application processing task" may include:

[0115] When the current content processing subtask is completed, a new current content processing subtask is determined from the media application processing tasks based on a preset execution order, and the processed content frame unit is used as a new content frame unit;

[0116] By reusing the memory block to be used, the step of executing the task processing logic of the current content processing subtask on the content frame unit is returned to obtain a final content frame unit that has completed all content processing subtasks in the media application processing task.

[0117] Among them, each content processing subtask is executed serially and has a certain execution order. Through memory reuse, all content processing subtasks can be executed serially one by one to complete the media application processing task.

[0118] The new current content processing subtask may be the next content processing subtask of the original current content processing subtask, and the output of the previous content processing subtask is the input of the next content processing subtask.

[0119] The final content frame unit is obtained by executing all content processing subtasks on the content frame unit.

[0120] In one possible implementation, in this embodiment, after the step of "reusing the to-be-used memory blocks based on a preset execution order to execute a new content processing subtask in the media application processing task on the processed content frame units, thereby obtaining a final content frame unit after executing all content processing subtasks in the media application processing task," the media content processing method may further include:

[0121] Determining an unprocessed content frame unit, where the unprocessed content frame unit is a content frame unit for which a media application processing task has not been executed;

[0122] By reusing the memory blocks to be used, each content processing subtask is performed on the unprocessed content frame unit to obtain a final content frame unit corresponding to the unprocessed content frame unit.

[0123] Among them, this embodiment can also provide a second level of memory reuse. The first level of memory reuse refers to memory reuse during the processing of each content processing subtask for the same content frame unit. The second level of memory reuse means that after a content frame unit completes the media application processing task, the memory block occupied by it during the processing process can be reused for other frames.

[0124] In one possible implementation, in this embodiment, the step of “reusing the to-be-used memory block to execute each content processing subtask on the unprocessed content frame unit to obtain a final content frame unit corresponding to the unprocessed content frame unit” may include:

[0125] Adding the memory address to be used corresponding to the memory block to be used to a preset free memory queue, the preset free memory queue is used to store the memory address of the memory block in the idle state;

[0126] Take out the memory address to be used from the preset free memory queue;

[0127] Each content processing subtask is executed on the unprocessed content frame unit through the unused memory block corresponding to the unused memory address to obtain a final content frame unit corresponding to the unprocessed content frame unit.

[0128] After a media application processing task of a content frame unit is completed using a memory block to be used, the memory block to be used can be marked as idle and added to a preset idle memory queue. The preset idle memory queue can be a FIFO (First In First Out) queue.

[0129] In some embodiments, memory blocks in the memory pool are marked with usage status information, and the usage status information includes an idle state and an occupied state;

[0130] After the step of "reusing the memory blocks to be used and executing the respective content processing subtasks on the content frame unit based on a preset execution order to obtain a final content frame unit", the following steps may also be performed:

[0131] When it is detected that the content frame unit has used up the memory block to be used, the usage status information of the memory block to be used is updated from the occupied state to the idle state;

[0132] Add the unused memory address of the unused memory block to a preset free memory queue;

[0133] Take out the memory address to be used from the preset free memory queue;

[0134] Through the memory block corresponding to the selected memory address to be used, based on the preset order, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain the final content frame unit. The unprocessed content frame unit is the content frame unit for which the media application processing task has not been executed.

[0135] In a specific scenario, when the media content to be processed is video image content, memory pools corresponding to multiple video resolutions can be preset, such as a memory pool corresponding to a video resolution of 8k, a memory pool corresponding to a resolution of 4k, etc. Each memory pool includes multiple memory blocks at the corresponding video resolution. If a certain video image content needs to be processed, the memory pool to be allocated can be determined based on the video resolution of the video image content. If there is no memory pool corresponding to the video resolution, memory can be applied for based on the image size, color sampling information, image pixel depth, and frame refresh frequency. The memory application size is M, and the memory pool applied for includes multiple memory blocks at the corresponding video resolution. The total size of all memory blocks is M.

[0136] When applying filters to image frames, if there is no temporal correlation between the frames, multiple frames can be processed in parallel using multiple memory blocks from the allocated memory pool. Note that temporal correlation refers to the temporal correlation between frames that require a reference frame for pixel resolution and the reference frame.

[0137] For each image frame unit, the filter processing flow includes super-resolution, color HDR, face, font, etc. The processing flow of each frame is to allocate a memory block for processing the frame, apply a certain filter to the frame (such as super-resolution), and then reuse the memory block for the next filter application (such as color HDR) until all filter applications are completed. The memory block can be marked as free and reused for other frames.

[0138] For example, the media application processing tasks of video images may include five content processing subtasks: super-resolution, color HDR (high dynamic range processing), face enhancement, text enhancement, and watermark removal. Each content processing subtask corresponds to a filter application. The output of the filter application corresponding to super-resolution is the input of the filter application corresponding to color HDR, the output of the filter application corresponding to color HDR is the input of the filter application corresponding to face enhancement, the output of the filter application corresponding to face enhancement is the input of the filter application corresponding to text enhancement, and the output of the filter application corresponding to text enhancement is the input of the filter application corresponding to watermark removal. For each image frame unit to be processed, a memory block to be used for processing the image frame unit can be allocated based on its content attribute information; then, the task processing logic of super-resolution is executed on the image frame unit through the memory block to be used. After the super-resolution content processing subtask is completed, the memory block to be used is not released, and the task processing logic of color HDR is executed on the image frame unit after super-resolution processing by reusing the memory block to be used; after the color HDR content processing subtask is completed, the memory block to be used is not released, and the image frame processed by color HDR is processed by reusing the memory block to be used. The unit performs the task processing logic for face enhancement. After completing the face enhancement content processing subtask, the unused memory block is not released, and the text enhancement task processing logic is performed on the image frame unit after the face enhancement processing by reusing the unused memory block. After completing the text enhancement content processing subtask, the unused memory block is not released, and the watermarking task processing logic is performed on the image frame unit after the text enhancement processing by reusing the unused memory block, obtaining the watermarked image frame unit. The watermarked image frame unit is the final image frame unit after completing all the content processing subtasks. Finally, if there are other image frame units that have not yet executed the media application processing task, the unused memory block can be provided to these unprocessed image frame units for reuse, so that the media application processing task can be performed on the unprocessed image frame units through the unused memory block.

[0139] Referring to Figure 1d, the specific process of performing five content processing subtasks of super-resolution, color HDR, face enhancement, text enhancement, and watermark removal on a certain image frame unit by reusing the memory blocks to be used is described as follows:

[0140] 1. Write the image frame unit into the memory block to be used. The module corresponding to the content processing subtask "super-resolution" reads the image frame unit from the memory block to be used, uses the image frame unit as the input of the module corresponding to "super-resolution", and processes the image frame unit through the task processing logic of the module corresponding to "super-resolution". When the content processing subtask corresponding to "super-resolution" is completed, the super-resolved image frame unit is obtained and output;

[0141] 2. The output super-resolution processed image frame unit is written into the memory block to be used. The module corresponding to the content processing subtask "Color HDR" reads the super-resolution processed image frame unit from the memory block to be used, uses the super-resolution processed image frame unit as the input of the module corresponding to "Color HDR", and processes the super-resolution processed image frame unit through the task processing logic of the module corresponding to "Color HDR". When the content processing subtask corresponding to "Color HDR" is completed, the image frame unit processed by Color HDR is obtained and output;

[0142] 3. The output image frame unit after color HDR processing is written into the memory block to be used. The module corresponding to the content processing subtask "face enhancement" reads the image frame unit after color HDR processing from the memory block to be used, uses the image frame unit after color HDR processing as the input of the module corresponding to "face enhancement", and processes the image frame unit after color HDR processing through the task processing logic of the module corresponding to "face enhancement". When the content processing subtask corresponding to "face enhancement" is completed, the image frame unit after face enhancement processing is obtained and output;

[0143] 4. The output image frame unit after face enhancement processing is written into the memory block to be used. The module corresponding to the content processing subtask "text enhancement" reads the image frame unit after face enhancement processing from the memory block to be used, uses the image frame unit after face enhancement processing as the input of the module corresponding to "text enhancement", and processes the image frame unit after face enhancement processing through the task processing logic of the module corresponding to "text enhancement". When the content processing subtask corresponding to "text enhancement" is completed, the image frame unit after text enhancement processing is obtained and output;

[0144] 5. The output image frame unit after text enhancement processing is written into the memory block to be used. The module corresponding to the content processing subtask "de-watermarking" reads the image frame unit after text enhancement processing from the memory block to be used, and uses the image frame unit after text enhancement processing as the input of the module corresponding to "de-watermarking". The image frame unit after text enhancement processing is processed through the task processing logic of the module corresponding to "de-watermarking". When the content processing subtask corresponding to "de-watermarking" is completed, the image frame unit after de-watermarking is obtained and output. The image frame unit after de-watermarking is the final image frame unit after executing all content processing subtasks.

[0145] The audio processing is similar to the video processing.

[0146] In another specific scenario, when the media content to be processed is audio content, a plurality of memory pools corresponding to audio sampling frequencies can be preset, such as a memory pool corresponding to an audio sampling frequency of 8k, a memory pool corresponding to an audio sampling frequency of 16k, etc. Each memory pool includes a plurality of memory blocks under the corresponding audio sampling frequencies. If it is necessary to process a certain audio content, it can be determined which memory pool to allocate based on the audio sampling frequency of the audio content. If there is no memory pool corresponding to the audio sampling frequency, memory can be applied for according to the audio sampling frequency and the audio sampling bit depth. The memory application size is recorded as N. The memory pool applied for includes a plurality of memory blocks under the corresponding audio sampling frequency, and the total size of all memory blocks is N.

[0147] When applying filters to audio frames, if there's no temporal correlation between the frames, multiple frames can be processed in parallel using multiple memory blocks from the allocated memory pool. Note that temporal correlation refers to the temporal correlation between frames that require a reference to another frame for parsing.

[0148] For each audio frame unit, the filter processing flow includes denoising, deburring, dejittering, compression repair, etc. The processing flow of each frame is to allocate a memory block for processing the frame, apply a certain filter to the frame, and then reuse the memory block for the next filter application after the processing is completed. After all filter applications are completed, the memory block can be marked as free and reused for other audio frame units.

[0149] For example, the media application processing task of audio may include four content processing sub-tasks: denoising, de-burring, de-jittering, and compression repair. Each content processing sub-task corresponds to a filter application. The output of the filter application corresponding to denoising is the input of the filter application corresponding to de-burring, the output of the filter application corresponding to de-burring is the input of the filter application corresponding to de-jittering, and the output of the filter application corresponding to de-jittering is the input of the filter application corresponding to compression repair. For each audio frame unit to be processed, a memory block to be used for processing this audio frame unit can be allocated according to its content attribute information; then, the task processing logic of denoising the audio frame unit is performed through the memory block to be used. After the content processing subtask of denoising is completed, the memory block to be used is not released, and the task processing logic of deburring the audio frame unit after denoising is performed by reusing the memory block to be used; after the content processing subtask of deburring is completed, the memory block to be used is not released, and the task processing logic of dejittering the audio frame unit after deburring is performed by reusing the memory block to be used; after the content processing subtask of dejitter is completed, the memory block to be used is not released, and the task processing logic of compression and repairing the audio frame unit after dejittering is performed by reusing the memory block to be used to obtain the compressed and repaired audio frame unit, which is the final audio frame unit after all content processing subtasks are executed. Finally, if there are other audio frame units that have not executed the media application processing task, the unused memory block can be provided to these unprocessed audio frame units for multiplexing, so as to execute the media application processing task on the unprocessed audio frame units through the unused memory block.

[0150] With reference to FIG1e , the specific process of performing the four content processing subtasks of denoising, deburring, dejittering, and compression repair on a certain audio frame unit by reusing the memory block to be used is described as follows:

[0151] 1. Write the audio frame unit to the unused memory block. The module corresponding to the content processing subtask "denoising" reads the audio frame unit from the unused memory block and uses the audio frame unit as the input of the module corresponding to "denoising". The audio frame unit is processed by the task processing logic of the module corresponding to "denoising". When the content processing subtask corresponding to "denoising" is completed, the denoised audio frame unit is obtained and output;

[0152] 2. The output denoised audio frame unit is written into the memory block to be used. The module corresponding to the content processing subtask "deburring" reads the denoised audio frame unit from the memory block to be used, uses the denoised audio frame unit as the input of the module corresponding to "deburring", and processes the denoised audio frame unit through the task processing logic of the module corresponding to "deburring". When the content processing subtask corresponding to "deburring" is completed, the denoised audio frame unit is obtained and output;

[0153] 3. The output de-glitched audio frame unit is written into the memory block to be used. The module corresponding to the content processing subtask "de-jitter" reads the de-glitched audio frame unit from the memory block to be used, uses the de-glitched audio frame unit as the input of the module corresponding to "de-jitter", and processes the de-glitched audio frame unit through the task processing logic of the module corresponding to "de-jitter". When the content processing subtask corresponding to "de-jitter" is completed, the de-glitched audio frame unit is obtained and output;

[0154] 4. Write the output de-jittered audio frame unit into the memory block to be used. The module corresponding to the content processing subtask "compression and repair" reads the de-jittered audio frame unit from the memory block to be used, uses the de-jittered audio frame unit as the input of the module corresponding to "compression and repair", and processes the de-jittered audio frame unit through the task processing logic of the module corresponding to "compression and repair". When the content processing subtask corresponding to "compression and repair" is completed, the compressed and repaired audio frame unit is obtained and output. The compressed and repaired audio frame unit is the final audio frame unit after executing all content processing subtasks.

[0155] In this embodiment, after the step of "reusing the to-be-used memory blocks based on the preset execution order to execute a new content processing subtask in the media application processing task on the processed content frame units, thereby obtaining a final content frame unit after executing all content processing subtasks in the media application processing task", the media content processing method may further include:

[0156] After completing the media application processing tasks of all content frame units, the processed media content to be processed is output, and the unused memory blocks occupied during the processing are released back to the memory pool. The processed media content to be processed includes each final content frame unit.

[0157] The processed media content to be processed can be used as input to the content encoding module for encoding, and then input to the mixed stream encapsulation module for container format encapsulation and compression, and then sent to the terminal.

[0158] In a specific scenario, as shown in FIG1f , which is a basic architecture diagram of the memory pool cache of the media content processing method provided by this application, the basic processing flow is as follows:

[0159] The filter processing flow of the avfilter apply effects application (the application effects processing module) takes as input the raw audio PCM and video YUV streams output by the decoding module. When allocating memory for the processed audio and video streams, the filter allocates memory pools corresponding to the audio sampling frequency and video resolution. It should be noted that audio and video processing are separate processes, with separate memory pools allocated for audio and video.

[0160] Among them, the memory pool can be allocated according to the audio sampling frequency (such as 8K, 16K, 24K, 32K, 44.1K...), video resolution (such as 8K, 4K, 2K, 1080P, 720P, 560P and 8bit, 10bit, 12bit, 420 / 422 / 444 color sampling information...), if the memory pool does not have the audio sampling rate and video resolution corresponding to the input video source, the size of the memory pool to be allocated can be determined according to its specific content attribute information. Specifically, the bit position (bit depth) can be applied for memory according to (8bit, 10bit, 12bit) / 8+1)byte. When applying for memory pool and dynamic memory, pre-apply according to audio sampling rate, video resolution, bit depth and color sampling information. For example, for an application scenario with an audio sampling rate of 8K, a video sampling method of YUV 4:2:2, an image pixel depth of 10bit, and a frame refresh rate of 50FPS, the memory pool size applied for audio and video for 2 seconds (i.e., the preset processing content duration in the above embodiment) is as follows:

[0161] The audio memory pool size can be recorded as PCM Buffersize, PCM Buffersize = audio sampling rate * audio sampling bit depth * duration = 8000 * 8bit * 2 = 128000bit

[0162] The video memory pool size can be recorded as YUV Buffersize, YUV Buffersize = image size * color sampling information * image pixel depth * duration * frame refresh rate = 7680 * 4320 * 1.5 * (10 / 8 + 1) * 8bit * 2 * 50 = 79,626,240,000bit

[0163] Among them, 1.5 means that in the YUV 4:2:2 sampling mode, the UV component is half the sampling of the Y component. When the number of bytes occupied by Y is 1, the number of bytes occupied by UV is 0.5.

[0164] In the case of only one frame of audio or video, or multiple frames with no temporal correlation between the frames, filter application processing can be performed in frame parallel or gop granularity parallel. The frames within the gop granularity can be temporally correlated.

[0165] For a certain content frame unit, the avfilter application can apply for a memory block from the memory pool as a frame buffer based on the decoded audio sampling rate and video resolution and frame rate. The avfilter application can reuse the buffer applied from the memory pool when processing in serial or parallel. The output of the previous Filter application is the input for the next Filter application. After the last Filter application is processed, the final content frame unit can be sent to the audio and video encoding core. Due to the different kernel memory layout architecture of the audio and video encoding core, it will reapply for memory and copy the output content of the last Filter. After the audio and video encoding core completes the copy, it can release the cache applied from the memory pool to the memory pool for reuse, that is, reclaim the occupied memory into the memory pool.

[0166] This application can provide a solution for improving media processing performance based on memory pool allocation. Specifically, it can improve the repeated memory application and release kernel calls during each Filter application processing in the existing framework by reusing memory pool buffer processing. The solution provided by this application can achieve an engineering performance improvement of 20%+ in a high-sampling rate and high-resolution audio and video application scenario (such as 8K, 44.1 7-channel application scenario).

[0167] As can be seen from the above, this embodiment can obtain media content to be processed, and the media content to be processed includes at least one content frame unit; according to the content attribute information of the media content to be processed, a memory pool for executing a media application processing task is allocated, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit in at least one content frame unit, a memory block to be used for processing the content frame unit is determined from the memory pool; the content processing subtask in the media application processing task is executed on the content frame unit through the memory block to be used; when the content processing subtask is completed, based on the preset execution order, the memory block to be used is reused to execute a new content processing subtask in the media application processing task on the processed content frame unit, and a final content frame unit is obtained after executing all the content processing subtasks in the media application processing task. The present application can execute each content processing subtask through memory reuse, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance.

[0168] According to the method described in the above embodiment, the following will be further described in detail by taking the example of the media content processing device being specifically integrated into the server.

[0169] The present application provides a method for processing media content. As shown in FIG2 , the specific process of the method for processing media content may be as follows:

[0170] 201. A server obtains media content to be processed, where the media content to be processed includes at least one content frame unit.

[0171] 202. The server obtains a preset mapping relationship set, where the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools.

[0172] The preset mapping relationship set may be a relationship table between preset content attributes and preset memory pools. Based on the content attribute information, the corresponding preset memory pool may be queried from the relationship table.

[0173] 203. The server searches the preset mapping relationship set for a preset memory pool corresponding to the content attribute information of the to-be-processed media content.

[0174] The memory pool for executing the media application processing task can be applied for allocation based on the content attribute information of the media content to be processed, which may be audio sampling frequency or video resolution.

[0175] 204. The server allocates a memory pool for executing a media application processing task based on the queried preset memory pool, where the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0176] In this embodiment, the media application processing task includes multiple links, each of which can be regarded as a content processing subtask. Different content processing subtasks can perform different types of content processing on content frame units. For example, the content processing subtask can be super-resolution processing, interpolation processing, or color enhancement processing.

[0177] Because memory allocation and release are both inefficient, this application can initially allocate a larger block of memory (i.e., a memory pool, which can be redistributed if this block is insufficient). Then, during the execution of tasks, the corresponding memory resources can be retrieved from this block of memory each time they are needed. Only after the entire media application processing task is completed will the memory block occupied during the processing be released to the operating system. This reduces the number of memory allocation and release operations, thereby improving efficiency.

[0178] The memory pool may include at least one memory block corresponding to content attribute information. The size of each memory block may correspond to the content attribute information of the media content to be processed (such as audio sampling rate, video resolution, etc.). The memory blocks in the memory pool are of the same size, so that the memory size allocated each time is fixed, avoiding the possibility of memory fragmentation.

[0179] In this embodiment, the media content processing method may further include:

[0180] When no preset memory pool corresponding to the content attribute information is found in the preset mapping relationship set, the memory application size is determined based on the content attribute information of the to-be-processed media content and the preset processing duration of the content;

[0181] Based on the memory request size, a memory pool is allocated for executing media application processing tasks.

[0182] If the media content to be processed is image content, the content attribute information may be the image size of the image frame unit, image color sampling information, image pixel depth, and frame refresh frequency.

[0183] Among them, if the media content to be processed is audio content, the content attribute information can be the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit. The audio sampling bit depth (bit depth) refers to the number of bits used to represent the amplitude at a certain point in time; the more bits there are, the more accurately the amplitude changes that can be represented.

[0184] 205. The server determines, for each content frame unit, from the memory pool a memory block to be used for processing the content frame unit.

[0185] In some embodiments, before the step of "determining a memory block to be used for processing the content frame unit from the memory pool", the following steps may also be included:

[0186] Set the preset free memory queue;

[0187] Storing the memory addresses of the free memory blocks in the memory pool in a preset free memory queue;

[0188] The step of “determining a memory block to be used for processing the content frame unit from the memory pool” may include:

[0189] Take out the memory address to be used from the preset free memory queue;

[0190] A memory block corresponding to the memory address to be used is determined as the memory block to be used for processing the content frame unit.

[0191] 206. The server executes each content processing subtask on the content frame unit based on the preset execution order by reusing the to-be-used memory block to obtain a final content frame unit.

[0192] In this embodiment, the step of "reusing the memory blocks to be used and executing the respective content processing subtasks on the content frame units based on a preset execution order to obtain the final content frame units" may include:

[0193] Determine a current content processing subtask from the media application processing tasks based on a preset execution order;

[0194] Execute the current content processing subtask on the content frame unit through the memory block to be used to obtain the processed content frame unit;

[0195] When the current content processing subtask is completed, a new current content processing subtask is determined from the media application processing tasks based on a preset execution order, and the processed content frame unit is used as a new content frame unit;

[0196] By reusing the memory block to be used, returning to execute the step of executing the current content processing subtask on the content frame unit, so as to obtain a final content frame unit.

[0197] Among them, each content processing subtask is executed serially and has a certain execution order. Through memory reuse, all content processing subtasks can be executed serially one by one to complete the media application processing task.

[0198] In this embodiment, after the step of "reusing the to-be-used memory blocks and executing the respective content processing subtasks on the content frame units based on a preset execution order to obtain the final content frame units", the media content processing method may further include:

[0199] Determining an unprocessed content frame unit, where the unprocessed content frame unit is a content frame unit for which a media application processing task has not been executed;

[0200] By reusing the memory blocks to be used, each content processing subtask is performed on the unprocessed content frame unit to obtain a final content frame unit corresponding to the unprocessed content frame unit.

[0201] 207. After completing the media application processing tasks for all content frame units, the server outputs the processed media content to be processed and releases the unused memory blocks occupied during the processing back to the memory pool. The processed media content to be processed includes the final content frame units.

[0202] In an embodiment of the present application, the processed media content to be processed can be used as the input of the content encoding module for encoding, and then the encoded media content to be processed can be input into the mixed stream encapsulation module for container format encapsulation and compression, and then sent to the terminal.

[0203] This application proposes a memory pool allocation solution that reuses the memory pool across all filter applications after the decoder module, reducing the engineering performance overhead caused by frequent memory allocation and release between filter applications. This can improve engineering performance by 20% or more in high-resolution, high-FPS video filter processing scenarios such as 4K and 8K. The solution provided by this application has been well-used in video cloud media processing products and has also been successfully applied in customers' 8K projects.

[0204] As can be seen from the above, this embodiment can obtain to-be-processed media content through a server, where the to-be-processed media content includes at least one content frame unit; obtain a preset mapping relationship set, where the preset mapping relationship set includes a mapping relationship between preset content attributes and a preset memory pool; query the preset mapping relationship set for a preset memory pool corresponding to the content attribute information of the to-be-processed media content; based on the queried preset memory pool, allocate a memory pool for executing a media application processing task, where the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit in at least one content frame unit, determine a to-be-used memory block from the memory pool for processing the content frame unit; by reusing the to-be-used memory blocks, execute each content processing subtask on the content frame unit based on the preset execution order to obtain a final content frame unit; after completing the media application processing tasks of all content frame units, output the processed to-be-processed media content, and release the to-be-used memory blocks occupied during the processing back to the memory pool, where the processed to-be-processed media content includes each final content frame unit.

[0205] This application can execute various content processing subtasks through memory reuse, and release the applied memory blocks back to the memory pool only after all media application processing tasks are completed. This can reduce the frequent memory allocation, application and release between various content processing subtasks, thereby improving media application processing performance.

[0206] To better implement the above method, an embodiment of the present application further provides a media content processing device. As shown in FIG3 , the media content processing device may include an acquisition unit 301, an allocation unit 302, a memory block determination unit 303, a first execution unit 304, and a second execution unit 305, as follows:

[0207] (1) Acquisition unit 301;

[0208] The acquisition unit is configured to acquire media content to be processed, where the media content to be processed includes at least one content frame unit.

[0209] (2) allocation unit 302;

[0210] An allocation unit is configured to allocate a memory pool for executing a media application processing task based on content attribute information of the media content to be processed, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0211] In a possible implementation, in some embodiments of the present application, the allocation unit may include an acquisition subunit, a query subunit, and an allocation subunit, as follows:

[0212] The acquisition subunit is configured to acquire a preset mapping relationship set, wherein the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools;

[0213] A query subunit, configured to query a preset memory pool corresponding to the content attribute information of the to-be-processed media content in the preset mapping relationship set;

[0214] The allocation subunit is configured to allocate a memory pool for executing media application processing tasks based on the queried preset memory pool.

[0215] In some embodiments of the present application, the allocation unit may further include an application size determination subunit and a memory pool allocation subunit, as follows:

[0216] The application size determination subunit is configured to determine a memory application size based on the content attribute information and a preset content processing duration when no preset memory pool corresponding to the content attribute information is found in the preset mapping relationship set;

[0217] The memory pool allocation subunit is used to allocate a memory pool for executing the media application processing task based on the memory application size.

[0218] In some embodiments of the present application, the media content to be processed includes image content, and the content frame unit is an image frame unit of the image content; the allocation unit may include a first calculation subunit, a first determination subunit, and a first allocation subunit, as follows:

[0219] The first calculation subunit is configured to calculate the memory capacity occupied by the image frame unit according to the image size, image color sampling information, and image pixel depth of the image frame unit;

[0220] A first determining subunit is configured to determine a memory application size according to the memory capacity and a frame refresh frequency corresponding to the image frame unit;

[0221] The first allocating subunit is configured to allocate a memory pool for executing a media application processing task on the image content based on the memory application size.

[0222] In some embodiments of the present application, the media content to be processed includes audio content, and the content frame unit is an audio frame unit of the audio content;

[0223] The allocation unit may include a second calculation subunit, a second determination subunit, and a second allocation subunit, as follows:

[0224] The second calculation subunit is configured to calculate a memory capacity occupied by the audio frame unit according to an audio sampling frequency and an audio sampling bit depth corresponding to the audio frame unit;

[0225] A second determining subunit is configured to determine a memory application size based on the memory capacity and a preset processing content duration;

[0226] The second allocating subunit is configured to allocate a memory pool for executing a media application processing task on the audio content based on the memory application size.

[0227] (3) memory block determination unit 303;

[0228] A memory block determining unit is configured to determine, for each content frame unit in the at least one content frame unit, a memory block to be used for processing the content frame unit from the memory pool.

[0229] (4) a first execution unit 304;

[0230] The first execution unit is configured to execute a content processing subtask in the media application processing task on the content frame unit by using the to-be-used memory block.

[0231] (5) second execution unit 305;

[0232] The second execution unit is used to reuse the memory block to be used to execute a new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain a final content frame unit after executing all content processing subtasks in the media application processing task.

[0233] In some embodiments of the present application, the first execution unit may include a task determination subunit and an execution subunit, and the second execution unit may include a redetermination subunit and a return subunit, as follows:

[0234] The task determination sub-unit is configured to determine a current content processing sub-task from the media application processing tasks based on the preset execution order;

[0235] an execution sub-unit, configured to execute a current content processing sub-task on the content frame unit through the to-be-used memory block to obtain a processed content frame unit;

[0236] a re-determining sub-unit, configured to, when the current content processing sub-task is completed, determine a new current content processing sub-task from the media application processing task based on the preset execution order, and use the processed content frame unit as a new content frame unit;

[0237] The return subunit is used to return to the step of executing the current content processing subtask on the content frame unit by reusing the memory block to be used, so as to obtain a final content frame unit that has completed executing all content processing subtasks in the media application processing task.

[0238] In some embodiments of the present application, the media content processing device may further include a multiplexing unit as follows:

[0239] The multiplexing unit is used to determine an unprocessed content frame unit, where the unprocessed content frame unit is a content frame unit that has not executed the media application processing task; by reusing the memory block to be used, each content processing subtask is executed on the unprocessed content frame unit to obtain a final content frame unit corresponding to the unprocessed content frame unit.

[0240] In some embodiments of the present application, the multiplexing unit may include an adding subunit, a removing subunit, and a task executing subunit, as follows:

[0241] The adding subunit is used to add the to-be-used memory address corresponding to the to-be-used memory block to a preset free memory queue, wherein the preset free memory queue is used to store the memory address of the memory block in the free state;

[0242] An extracting subunit, configured to extract the memory address to be used from the preset free memory queue;

[0243] The task execution sub-unit is configured to execute each content processing sub-task on the unprocessed content frame unit through the to-be-used memory block corresponding to the to-be-used memory address, to obtain a final content frame unit corresponding to the unprocessed content frame unit.

[0244] In some embodiments of the present application, the media content processing device may further include an output unit as follows:

[0245] The output unit is configured to output the processed media content to be processed after completing the media application processing tasks of all content frame units, and release the unused memory blocks occupied during the processing back to the memory pool, wherein the processed media content to be processed includes each final content frame unit.

[0246] As can be seen from the above, in this embodiment, the acquisition unit 301 can acquire the media content to be processed, and the media content to be processed includes at least one content frame unit; the allocation unit 302 allocates a memory pool for executing the media application processing task according to the content attribute information of the media content to be processed, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; the memory block determination unit 303 determines the memory block to be used for processing the content frame unit from the memory pool for each content frame unit; the first execution unit 304 executes the content processing subtask in the media application processing task on the content frame unit through the memory block to be used; the second execution unit 305 reuses the memory block to be used to execute the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is executed, and obtains the final content frame unit after executing all the content processing subtasks in the media application processing task. The present application can execute each content processing subtask through memory reuse, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance.

[0247] The present application also provides an electronic device, as shown in FIG4 , which shows a schematic diagram of the structure of the electronic device involved in the present application embodiment. The electronic device may be a terminal or a server, etc. Specifically:

[0248] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that the electronic device structure shown in FIG4 does not limit the electronic device and may include more or fewer components than shown, or combine certain components, or arrange the components differently.

[0249] Processor 401 is the control center of the electronic device. It connects all parts of the electronic device using various interfaces and circuits. It performs various functions of the electronic device and processes data by running or executing software programs and / or modules stored in memory 402 and accessing data stored in memory 402. Optionally, processor 401 may include one or more processing cores. Preferably, processor 401 may integrate an application processor and a modem processor. The application processor primarily handles the operating system, user interface, and application programs, while the modem processor primarily handles wireless communications. It is understood that the modem processor may not be integrated into processor 401.

[0250] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage device. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0251] The electronic device also includes a power supply 403 for supplying power to various components. Preferably, the power supply 403 can be logically connected to the processor 401 via a power management system, thereby enabling the power management system to manage charging, discharging, and power consumption. The power supply 403 can also include one or more DC or AC power supplies, a recharging system, a power failure detection circuit, a power converter or inverter, a power status indicator, and other arbitrary components.

[0252] The electronic device may further include an input unit 404, which may be configured to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal inputs related to user settings and function control.

[0253] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically, in this embodiment, the processor 401 in the electronic device loads the executable files corresponding to one or more application processes into the memory 402 according to the following instructions, and the processor 401 runs the application stored in the memory 402, thereby implementing the various functions provided in the above embodiments.

[0254] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0255] Those skilled in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be accomplished by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0256] To this end, an embodiment of the present application provides a computer-readable storage medium, which stores a plurality of instructions, and the instructions can be loaded by a processor to execute the steps in any one of the media content processing methods provided in the embodiments of the present application.

[0257] The specific implementation of the above operations can be found in the previous embodiments and will not be repeated here.

[0258] The computer-readable storage medium may include a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0259] Since the instructions stored in the computer-readable storage medium can execute the steps of any media content processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any media content processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0260] According to one aspect of the present application, a computer program product is provided, comprising a computer program stored in a computer-readable storage medium. A processor of a computer device reads the computer program from the computer-readable storage medium and executes the computer program, causing the computer device to perform the methods provided in various optional implementations of the aforementioned media content processing aspects.

[0261] The above is a detailed introduction to a media content processing method and related equipment provided in an embodiment of the present application. Specific examples are used herein to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core ideas. At the same time, for those skilled in the art, according to the ideas of the present application, there may be changes in the specific implementation methods and application scope. In summary, the content of this specification should not be understood as limiting the present application.

Claims

1. A media content processing method, the method being executed by an electronic device, comprising: Acquire media content to be processed, where the media content to be processed includes at least one content frame unit; Allocate a memory pool for executing a media application processing task according to content attribute information of the to-be-processed media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; For each content frame unit in the at least one content frame unit, determining a memory block to be used for processing the content frame unit from the memory pool; Executing a content processing subtask in the media application processing task on the content frame unit through the memory block to be used; When the content processing subtask is completed, based on the preset execution order, the memory block to be used is reused to execute a new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain a final content frame unit that has executed all the content processing subtasks in the media application processing task.

2. The method according to claim 1, wherein the step of executing the content processing subtask in the media application processing task on the content frame unit through the to-be-used memory block comprises: Based on the preset execution order, determining a current content processing subtask from the media application processing tasks; Execute the current content processing subtask on the content frame unit through the memory block to be used to obtain a processed content frame unit; When the content processing subtask is completed, based on the preset execution order, the memory block to be used is reused to execute a new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain a final content frame unit after executing all content processing subtasks in the media application processing task, including: When the current content processing subtask is completed, based on the preset execution order, a new current content processing subtask is determined from the media application processing task, and the processed content frame unit is used as a new content frame unit; By reusing the memory block to be used, returning to the step of executing the current content processing subtask on the content frame unit, a final content frame unit is obtained after all content processing subtasks in the media application processing task are executed.

3. The method according to claim 1, further comprising: reusing the to-be-used memory block based on the preset execution order to execute a new content processing subtask in the media application processing task on the processed content frame unit to obtain a final content frame unit after executing all content processing subtasks in the media application processing task; Determining an unprocessed content frame unit, wherein the unprocessed content frame unit is a content frame unit that does not execute the media application processing task; By reusing the memory block to be used, each content processing subtask is performed on the unprocessed content frame unit to obtain a final content frame unit corresponding to the unprocessed content frame unit.

4. The method according to claim 3, wherein the performing of each content processing subtask on the unprocessed content frame unit by reusing the to-be-used memory block to obtain a final content frame unit corresponding to the unprocessed content frame unit comprises: Adding the to-be-used memory address corresponding to the to-be-used memory block to a preset free memory queue, wherein the preset free memory queue is used to store the memory address of the memory block in an idle state; Taking out the memory address to be used from the preset free memory queue; Each content processing subtask is performed on the unprocessed content frame unit through the unused memory block corresponding to the unprocessed memory address to obtain a final content frame unit corresponding to the unprocessed content frame unit.

5. The method according to claim 1, allocating a memory pool for executing a media application processing task according to content attribute information of the to-be-processed media content, comprises: Obtaining a preset mapping relationship set, wherein the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools; Querying a preset memory pool corresponding to the content attribute information of the to-be-processed media content in the preset mapping relationship set; Based on the queried preset memory pool, a memory pool for executing the media application processing task is allocated.

6. The method according to claim 5, further comprising: When no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set, determining a memory application size based on the content attribute information and a preset content processing duration; Based on the memory application size, a memory pool for executing the media application processing task is allocated.

7. The method according to claim 1, wherein the media content to be processed comprises image content, and the content frame unit is an image frame unit of the image content; The allocating a memory pool for executing a media application processing task according to content attribute information of the to-be-processed media content includes: Calculating the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel bit depth of the image frame unit; Determining a memory application size according to the memory capacity and the frame refresh frequency corresponding to the image frame unit; Based on the memory application size, a memory pool is allocated for executing a media application processing task on the image content.

8. The method according to claim 1, wherein the media content to be processed comprises audio content, and the content frame unit is an audio frame unit of the audio content; The allocating a memory pool for executing a media application processing task according to content attribute information of the to-be-processed media content includes: Calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and the audio sampling bit depth corresponding to the audio frame unit; Determine the memory application size according to the memory capacity and the preset processing content duration; Based on the memory application size, a memory pool is allocated for executing a media application processing task on the audio content.

9. The method according to claim 1, further comprising: reusing the to-be-used memory block based on the preset execution order to execute a new content processing subtask in the media application processing task on the processed content frame unit to obtain a final content frame unit after executing all content processing subtasks in the media application processing task; After completing the media application processing tasks of all content frame units, the processed media content to be processed is output, and the unused memory blocks occupied during the processing are released back to the memory pool, wherein the processed media content to be processed includes each final content frame unit.

10. A media content processing device, the device being deployed on an electronic device, comprising: An acquisition unit, configured to acquire media content to be processed, wherein the media content to be processed includes at least one content frame unit; an allocating unit, configured to allocate a memory pool for executing a media application processing task according to content attribute information of the media content to be processed, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; a memory block determining unit, configured to determine, for each content frame unit in the at least one content frame unit, a memory block to be used for processing the content frame unit from the memory pool; A first execution unit, configured to execute a content processing subtask in the media application processing task on the content frame unit through the to-be-used memory block; The second execution unit is used to reuse the memory block to be used to execute a new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain a final content frame unit that completes the execution of all content processing subtasks in the media application processing task.

11. An electronic device comprising a memory and a processor; the memory stores a plurality of instructions, and the processor is used to run the plurality of instructions in the memory to execute the steps in the media content processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium storing a plurality of instructions, wherein the instructions are suitable for being loaded by a processor to execute the steps in the media content processing method according to any one of claims 1 to 9.

13. A computer program product, comprising a computer program, wherein when the computer program is executed by a processor, the steps in the media content processing method according to any one of claims 1 to 9 are implemented.

Images