Media content processing method and related equipment

By allocating memory pools and multiplexing memory blocks in media content processing, the performance problems caused by frequent memory allocation and release in the prior art are solved, and the efficiency of media application processing is improved.

CN119088534APending Publication Date: 2024-12-06TENCENT TECHNOLOGY (SHENZHEN) CO LTD
View PDF 0 Cites 1 Cited by

Patent Information

Application Number
CN202310660301.6
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-06-05
Publication Date
2024-12-06

AI Technical Summary

Technical Problem

The processing process of each media application in the existing media content processing framework is independent, resulting in frequent memory allocation and release, resulting in poor performance.

Method used

By allocating memory pools and multiplexing memory blocks on it, content processing subtasks in media application processing tasks are performed for each content frame unit, reducing memory allocation and release operations.

Benefits of technology

Improves media application processing performance, reduces the number of memory requests and releases, and improves processing efficiency.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN119088534A_ABST
    Figure CN119088534A_ABST
Patent Text Reader

Abstract

The invention discloses a media content processing method and related equipment. A memory pool used for executing a media application processing task can be allocated according to content attribute information of to-be-processed target media content, and the media application processing task comprises at least one content processing sub-task arranged according to a preset execution sequence; for each content frame unit, determining a target memory block for processing the content frame unit from a memory pool; executing a content processing subtask in the media application processing task on the content frame unit through the target memory block; and when the execution of the content processing subtasks is completed, reusing the target memory block to execute a new content processing subtask in the media application processing task based on a preset execution sequence, and obtaining a target processed content frame unit after all the content processing subtasks in the media application processing task are executed. According to the method and the device, frequent memory allocation application and release among the content processing subtasks can be reduced, so that the media application processing performance can be improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of computer technology, and in particular to a media content processing method and related equipment. Background Art

[0002] With the development of computer technology and mobile communication technology, various new media platforms have emerged, and more and more users are releasing various multimedia content (such as short videos, etc.) through various new media platforms. In order to ensure the quality of multimedia content, it is necessary to perform various types of application processing on multimedia content, such as image quality enhancement, intelligent editing, etc., among which image quality enhancement can include frame insertion, text enhancement, denoising, color enhancement, compression repair and other processing, and intelligent editing can include intelligent subtitle addition, real-time editing of live broadcasts, watermark removal, video screenshots, etc.

[0003] Specifically, the basic operations of multimedia content processing may include content file input, decapsulation, decoding, content encoding, container format packaging and packaging, content rendering, and various types of media application processing. In the current media content processing framework, these media application processings for processing media content are independent, and each media application processing flow is a self-closed loop. Each media application processing will re-allocate and release memory, that is, each media application processing is independent in memory allocation, and each media application processing will independently apply for memory allocation and recovery, which will lead to frequent memory access allocation and release, resulting in poor media application processing performance. Summary of the invention

[0004] An embodiment of the present application provides a media content processing method and related equipment. The related equipment may include a media content processing device, an electronic device, a computer-readable storage medium and a computer program product, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance.

[0005] The present application provides a method for processing media content, including:

[0006] Acquire target media content to be processed, wherein the target media content includes at least one content frame unit;

[0007] Allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information;

[0008] For each content frame unit, determining a target memory block for processing the content frame unit from the memory pool;

[0009] Executing the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block;

[0010] When the content processing subtask is completed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

[0011] Accordingly, an embodiment of the present application provides a media content processing device, including:

[0012] An acquisition unit, configured to acquire target media content to be processed, wherein the target media content includes at least one content frame unit;

[0013] an allocating unit, configured to allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information;

[0014] A memory block determination unit, configured to determine, for each content frame unit, a target memory block for processing the content frame unit from the memory pool;

[0015] A first execution unit, configured to execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block;

[0016] The second execution unit is used to reuse the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

[0017] Optionally, in some embodiments of the present application, the first execution unit may include a task determination subunit and an execution subunit, and the second execution unit may include a redetermination subunit and a return subunit, as follows:

[0018] The task determination sub-unit is used to determine the target content processing sub-task to be processed currently from the media application processing tasks based on the preset execution order;

[0019] An execution sub-unit, configured to execute the task processing logic of the target content processing sub-task on the content frame unit through the target memory block to obtain a processed content frame unit;

[0020] Re-determining a sub-unit, for determining a new current target content processing sub-task to be processed from the media application processing task based on the preset execution order when the target content processing sub-task is completed, and using the processed content frame unit as a new content frame unit;

[0021] The return subunit is used to return to the step of executing the task processing logic of executing the target content processing subtask on the content frame unit by reusing the target memory block, so as to obtain the target processed content frame unit after executing all content processing subtasks in the media application processing task.

[0022] Optionally, in some embodiments of the present application, the media content processing device may further include a multiplexing unit as follows:

[0023] The multiplexing unit is used to determine an unprocessed content frame unit, where the unprocessed content frame unit is a content frame unit that has not executed the media application processing task; by reusing the target memory block, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain a target processed content frame unit corresponding to the unprocessed content frame unit.

[0024] Optionally, in some embodiments of the present application, the multiplexing unit may include an adding subunit, a removing subunit, and a task executing subunit, as follows:

[0025] The adding subunit is used to add the target memory address corresponding to the target memory block to a preset free memory queue, and the preset free memory queue is used to store the memory addresses of the memory blocks in the free state;

[0026] A fetching subunit, used for fetching the target memory address from the preset free memory queue;

[0027] The task execution subunit is used to execute the task processing logic of each content processing subtask on the unprocessed content frame unit through the target memory block corresponding to the target memory address, so as to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

[0028] Optionally, in some embodiments of the present application, the allocation unit may include an acquisition subunit, a query subunit and an allocation subunit, as follows:

[0029] The acquisition subunit is used to acquire a preset mapping relationship set, wherein the preset mapping relationship set includes a mapping relationship between a preset content attribute and a preset memory pool;

[0030] A query subunit, configured to query a preset memory pool corresponding to the content attribute information of the target media content in the preset mapping relationship set;

[0031] The allocation subunit is used to allocate a memory pool for executing the media application processing task based on the queried preset memory pool.

[0032] Optionally, in some embodiments of the present application, the allocation unit may further include an application size determination subunit and a memory pool allocation subunit, as follows:

[0033] The application size determination subunit is used to determine the memory application size based on the content attribute information of the target media content and the preset processing content duration when no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set;

[0034] The memory pool allocation subunit is used to allocate a memory pool for executing the media application processing task based on the memory application size.

[0035] Optionally, in some embodiments of the present application, the target media content includes image content, the content frame unit is an image frame unit of the image content; the allocation unit may include a first calculation subunit, a first determination subunit and a first allocation subunit, as follows:

[0036] The first calculation subunit is used to calculate the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel depth of the image frame unit;

[0037] A first determining subunit, configured to determine a memory application size according to the memory capacity and a frame refresh frequency corresponding to the image frame unit;

[0038] The first allocation subunit is used to allocate a memory pool for executing a media application processing task on the image content based on the memory application size.

[0039] Optionally, in some embodiments of the present application, the target media content includes audio content, and the content frame unit is an audio frame unit of the audio content;

[0040] The allocation unit may include a second calculation subunit, a second determination subunit, and a second allocation subunit, as follows:

[0041] The second calculation subunit is used to calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and the audio sampling bit depth corresponding to the audio frame unit;

[0042] A second determining subunit is used to determine the memory application size according to the memory capacity and the preset processing content duration;

[0043] The second allocation subunit is used to allocate a memory pool for executing a media application processing task on the audio content based on the memory application size.

[0044] Optionally, in some embodiments of the present application, the media content processing device may further include an output unit as follows:

[0045] The output unit is used to output the processed target media content after completing the media application processing tasks of all content frame units, and release the target memory blocks occupied during the processing back to the memory pool, wherein the processed target media content includes each target processed content frame unit.

[0046] An electronic device provided in an embodiment of the present application includes a processor and a memory, wherein the memory stores a plurality of instructions, and the processor loads the instructions to execute the steps in the media content processing method provided in the embodiment of the present application.

[0047] The embodiment of the present application further provides a computer-readable storage medium having a computer program stored thereon, wherein the computer program, when executed by a processor, implements the steps in the media content processing method provided in the embodiment of the present application.

[0048] In addition, an embodiment of the present application also provides a computer program product, including a computer program or instructions, which, when executed by a processor, implements the steps in the media content processing method provided in the embodiment of the present application.

[0049] The embodiment of the present application provides a media content processing method and related equipment; the target media content to be processed can be obtained, and the target media content includes at least one content frame unit; according to the content attribute information of the target media content, a memory pool for executing a media application processing task is allocated, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, a target memory block for processing the content frame unit is determined from the memory pool; through the target memory block, the task processing logic of the content processing subtask in the media application processing task is executed on the content frame unit; when the content processing subtask is executed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and the target processed content frame unit after executing all the content processing subtasks in the media application processing task is obtained. The present application can execute each content processing subtask through memory reuse, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance. BRIEF DESCRIPTION OF THE DRAWINGS

[0050] In order to more clearly illustrate the technical solutions in the embodiments of the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For those skilled in the art, other drawings can be obtained based on these drawings without creative work.

[0051] Figure 1a is a schematic diagram of a scenario of a media content processing method provided in an embodiment of the present application;

[0052] Figure 1b is a flow chart of a media content processing method provided by an embodiment of the present application;

[0053] Figure 1c It is a framework diagram of the media content processing method provided in the embodiment of the present application;

[0054] Figure 1d is an illustration of a media content processing method provided by an embodiment of the present application;

[0055] Figure 1e is another illustration of the media content processing method provided by an embodiment of the present application;

[0056] Figure 1f is another framework diagram of the media content processing method provided by an embodiment of the present application;

[0057] Figure 2is another flow chart of the media content processing method provided by an embodiment of the present application;

[0058] Figure 3 is a structural diagram of a media content processing device provided in an embodiment of the present application;

[0059] Figure 4 It is a schematic diagram of the structure of an electronic device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative work are within the scope of protection of this application.

[0061] The embodiment of the present application provides a media content processing method and related equipment, and the related equipment may include a media content processing device, an electronic device, a computer-readable storage medium and a computer program product. The media content processing device may be integrated in an electronic device, and the electronic device may be a terminal or a server.

[0062] It is understandable that the media content processing method of this embodiment can be executed on a terminal, can be executed on a server, or can be executed by both the terminal and the server. The above examples should not be construed as limiting the present application.

[0063] like Figure 1a As shown, a terminal and a server jointly execute a media content processing method as an example. The media content processing system provided in the embodiment of the present application includes a terminal 10 and a server 11, etc. The terminal 10 and the server 11 are connected via a network, such as a wired or wireless network connection, etc., wherein the media content processing device can be integrated in the server.

[0064] The server 11 can be used to: obtain target media content to be processed, the target media content includes at least one content frame unit; allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, determine a target memory block for processing the content frame unit from the memory pool; execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; when the content processing subtask is completed, reuse the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order, and obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task; after completing the media application processing tasks of all the content frame units, output the processed target media content, and release the target memory blocks occupied during the processing back to the memory pool, the processed target media content includes each target processed content frame unit. The server 11 may be an independent physical server, a server cluster or a distributed system composed of multiple physical servers, or a cloud server providing cloud computing services.

[0065] The terminal 10 may be used to receive the processed target media content output by the server 11, and render and display the processed target media content. The terminal 10 may include a mobile phone, an intelligent voice interaction device, an intelligent home appliance, a vehicle terminal, an aircraft, a tablet computer, a notebook computer, or a personal computer (PC). A client may also be set on the terminal 10, and the client may be an application client or a browser client, etc.

[0066] The media content processing method provided in the embodiment of the present application relates to cloud computing in the field of cloud technology.

[0067] Among them, cloud technology refers to a hosting technology that unifies hardware, software, network and other resources in a wide area network or local area network to realize data computing, storage, processing and sharing. Cloud technology is a general term for network technology, information technology, integration technology, management platform technology, application technology, etc. based on the cloud computing business model application. It can form a resource pool, which is used on demand and flexible and convenient. Cloud computing technology will become an important support. The background services of the technical network system require a large amount of computing and storage resources, such as video websites, picture websites and more portal websites. With the high development and application of the Internet industry, in the future, each item may have its own identification mark, and all need to be transmitted to the background system for logical processing. Data of different levels will be processed separately. All kinds of industry data require strong system backing support, which can only be achieved through cloud computing.

[0068] Among them, cloud computing refers to the delivery and use mode of IT infrastructure, which means obtaining required resources through the network in an on-demand and easily scalable manner; in a broad sense, cloud computing refers to the delivery and use mode of services, which means obtaining required services through the network in an on-demand and easily scalable manner. This service can be related to IT and software, the Internet, or other services. Cloud computing is the product of the development and integration of traditional computer and network technologies such as grid computing, distributed computing, parallel computing, utility computing, network storage technologies, virtualization, and load balancing. With the development of the Internet, real-time data streams, and the diversification of connected devices, as well as the promotion of search services, social networks, mobile commerce, and open collaboration, cloud computing has developed rapidly. Different from the previous parallel distributed computing, the emergence of cloud computing will promote revolutionary changes in the entire Internet model and enterprise management model in terms of concept.

[0069] It should be noted that the description order of the following embodiments is not intended to limit the preferred order of the embodiments.

[0070] This embodiment will be described from the perspective of a media content processing device. The media content processing device may be integrated into an electronic device, which may be a server or a terminal.

[0071] like Figure 1b As shown, the specific process of the media content processing method may be as follows:

[0072] 101. Acquire target media content to be processed, where the target media content includes at least one content frame unit.

[0073] The target media content may be a video type of content, which may include image content and audio content. In some embodiments, the target media content may also include only audio content or image content, which is not specifically limited in this embodiment. The target media content to be processed may be media content that requires a media application processing task, and may specifically be media content that has been decapsulated and decoded.

[0074] A content frame unit may include only one content frame or multiple content frames, such as a content frame unit may include all frames in a gap (GOP) between two I frames (key frames). GOP refers to the time interval between two I frames in video encoding.

[0075] Specifically, if the target media content is image content, the content frame unit may be an image frame unit; if the target media content is audio content, the content frame unit may be an audio frame unit.

[0076] In specific application scenarios, such as Figure 1c As shown in the figure, it is a basic architecture diagram of the media processing framework. The basic process of multimedia content processing may include content file input (Input File), decapsulation (demuxing), decoding (decoding), various types of media application processing, content encoding (encoding), container format encapsulation and packaging, and content rendering. The content files here can be audio and video files.

[0077] Among them, the input of the decapsulation module is the Input File source container format (such as mp4, mp3, ts, FLV, etc.), and the output can be AAC, Opus, H.264, H.265, H.266, AV1 and other video raw streams. The output audio of the decoding module is PCM, and the video is YUV raw stream. The input and output audio of the avfilter apply effects application (application effect processing module, that is, the above-mentioned media application processing) is PCM, and the video is YUV. The input audio of the content encoding module is PCM, and the video is YUV, and the output is the encoded and compressed raw stream such as AAC, Opus, H.264, H.265, H.266, AV1 and other audio and video compression. The input of the container format encapsulation and packaging module (that is, the mixed stream encapsulation module) is AAC, Opus, H.264, H.265, H.266, AV1 and other audio and video compressed raw streams, and the output is container encapsulated such as mp4, mp3, ts, FLV, etc.

[0078] The container format encapsulation and packaging module outputs a standard container format such as mp4, mov, ts, AAC, etc. The audio and video package obtained by encapsulation and packaging in the container format can be sent to a terminal. After receiving the audio and video package, the terminal can decompress it according to the standard container format, and then play it after decoding and rendering. The terminal here can include mobile phones, computers, controllers, televisions, virtual reality display devices, etc., which are not limited in this embodiment.

[0079] Among them, PCM (Pulse Code Modulation) is a method of digitizing analog signals. YUV (Y stands for Luminance or Luma, brightness; U and V stand for Chrominance or Chroma, chrominance) is a color encoding system, mainly used in video and graphics processing pipelines.

[0080] The decapsulation module can separate the input encapsulation format data into audio stream compressed coded data and video stream compressed coded data. There are many types of encapsulation formats, such as MP4, MKV, RMVB, TS, FLV, AVI, etc. Its function is to put the compressed and coded video data and audio data together in a certain format. For example, after the decapsulation operation, the FLV format data outputs the H.264 encoded video stream and the AAC encoded audio stream.

[0081] The decoding module can decode the video / audio compression encoded data into uncompressed video / audio raw data. Through decoding, the compressed encoded video data is output as uncompressed color data, such as YUV420P, RGB, etc.; the compressed encoded audio data is output as uncompressed audio sampling data, such as PCM data.

[0082] Among them, the input and output audio format of the avfilter apply effects application processing flow is PCM, and the video format is YUV. It can perform audio and video quality enhancement and intelligent editing processing according to audio and video metadata such as audio sampling frequency and video resolution. For example, image quality enhancement may include super-resolution, interpolation, color HDR (high dynamic range processing), face enhancement, text enhancement, denoising, deburring, de-jittering, color enhancement, compression repair, audit / quality inspection, and high-resistance digital watermarking; intelligent editing processing may include intelligent orchestration pipelines, intelligent subtitles, live real-time editing, splicing, abnormal container / audio and video repair, highlights, watermark removal, recording / screenshot / mixing, digital twins, etc. The main focus of this application is the avfilter apply effects application. The avfilter apply effects application can specifically include Filter applications such as super-resolution, interpolation, enhancement, color HDR, watermark, splicing, highlights, recording screenshots, and mixing (one Filter application corresponds to one content processing subtask). The processing of these Filter applications is independent, and the input and output are both audio PCM and video YUV. In the current related technologies, each Filter application is a self-closed loop, and each Filter application will re-allocate and release memory during processing. Existing Filter processing frameworks such as FFmpeg, Gstreamer, OpenCV, etc. are independent in memory allocation and do not reuse memory. Each application processing will independently re-apply for allocation and recovery of memory, which will lead to frequent memory allocation and recovery. Frequent memory access allocation and release will lead to inefficient filter application processing performance. The memory performance of this type of framework architecture in high sampling rate and high resolution application scenarios of audio and video (such as 4K, 8K and other application scenarios) is increasingly becoming a bottleneck for real-time application processing.

[0083] Among them, Filter is a filter that can provide many audio and video special effects processing functions, such as video scaling, interception, flipping, overlaying, etc.

[0084] Among them, HDR (High-Dynamic Range) can provide a wider color range and image details compared to ordinary images, improve the image light and dark contrast, greatly restore the real environment, and present extremely high image quality.

[0085] The audio sampling frequency (sampling rate) refers to how many samples the analog signal takes per unit time by the recording device. The higher the sampling frequency, the more real and natural the mechanical wave waveform will be. On today's mainstream acquisition cards, the sampling frequency is generally divided into five levels: 8000Hz, 11025Hz, 22050Hz, 24000Hz, 44100Hz, and 48000Hz. 11025Hz can achieve the sound quality of AM broadcasting, while 22050Hz and 24000HZ can achieve the sound quality of FM broadcasting. 44100Hz is the theoretical limit of CD sound quality, and 48000Hz is more precise.

[0086] Resolution is defined as the number of pixels organized or arranged by width and height on a screen or image. Specifically, resolution measures the number of pixels an image has, and the more pixels, the higher the quality. The most commonly used video resolutions are called "formats," technically known as ultra-high definition or full high definition. Common video resolutions include: 480P (640x 480), 720P (1280x720), 1080P (1920x 1080), 4K (3840x 2160), and 8K (7680x 4320).

[0087] This application can propose a memory pool allocation solution, which reuses the memory pool when processing all filter applications behind the decoder decoding module, reducing the engineering performance consumption caused by frequent memory allocation applications and releases between filter applications, which can improve the engineering performance by 20%+ in high-resolution, high-FPS video filter processing application scenarios such as 4K and 8K. The solution provided by this application has a good application in video cloud media processing business products and is also well used in customers' 8K projects.

[0088] Among them, FPS stands for "frame refresh rate (in Hz)". FPS is a definition in the field of images, which refers to the number of frames transmitted per second. Generally speaking, it refers to the number of frames of animation or video. FPS measures the amount of information used to save and display dynamic videos. The more frames per second, the smoother the displayed action will be.

[0089] 102. Allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0090] Among them, the media application processing task is the task of performing media processing on each content frame unit of the target media content. Media processing (MPS) can specifically refer to container format decapsulation, decoding, enhancement and application processing (such as superimposing watermarks, video screenshots, smart covers, smart editing, image quality enhancement, etc.), audio and video encoding format transcoding, packaging and distribution processing, etc.

[0091] The memory pool for executing the media application processing task can be applied for allocation according to the content attribute information of the target media content, where the content attribute information may be audio sampling frequency or video resolution, etc.

[0092] Among them, memory pool is a memory allocation method, also known as fixed-size-blocks allocation. In the current related technologies, memory allocation is usually directly applied using APIs (Application Programming Interface) such as new and malloc. The disadvantage of this is that since the size of the memory block applied is uncertain, a large amount of memory fragmentation will be caused when it is frequently used. A large amount of fragmentation will result in memory release, cleaning and merging when the memory is insufficient, which will consume a large amount of memory defragmentation and reduce performance.

[0093] Since both the application and release of memory are inefficient, the present application can apply for a larger block of memory (i.e., a memory pool, specifically, when the block of memory is insufficient, memory can be redistributed) at the beginning, and then, in the process of executing tasks, the corresponding memory resources can be taken out from this block of memory each time when needed. And after all the media application processing tasks are completed, the memory blocks occupied in the processing process are actually released to the operating system, which can reduce the number of memory application and release operations, thereby improving efficiency.

[0094] The memory pool may include at least one memory block corresponding to content attribute information, and the size of each memory block may correspond to the content attribute information of the target media content (such as audio sampling rate, video resolution, etc.). Specifically, the sizes of the memory blocks in the memory pool are the same, so that the size of the memory allocated each time is fixed, avoiding the possibility of memory fragmentation.

[0095] In this embodiment, the media application processing task includes multiple links, each of which can be regarded as a content processing subtask. Different content processing subtasks can perform different types of content processing on content frame units. For example, the content processing subtask can be super-resolution processing, interpolation processing, or color enhancement processing.

[0096] Optionally, in this embodiment, the step of “allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content” may include:

[0097] Obtaining a preset mapping relationship set, wherein the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools;

[0098] Querying a preset memory pool corresponding to the content attribute information of the target media content in the preset mapping relationship set;

[0099] Based on the queried preset memory pool, a memory pool for executing the media application processing task is allocated.

[0100] The preset mapping relationship set may be a relationship table between preset content attributes and preset memory pools. Based on the content attribute information, the corresponding preset memory pool may be queried from the relationship table.

[0101] The queried preset memory pool may be determined as a memory pool for executing the media application processing task.

[0102] In some embodiments, the memory resources of the memory pool can be pre-divided into at least one memory block corresponding to the content attribute information (such as the frame resolution size), so as to improve the memory allocation efficiency. For example, the memory pool corresponding to the video resolution 8K can divide its memory resources based on the video resolution 8K to obtain at least one memory block corresponding to the video resolution 8K.

[0103] Optionally, in this embodiment, the media content processing method may further include:

[0104] When no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set, determining a memory application size based on the content attribute information of the target media content and a preset content processing duration;

[0105] Based on the memory application size, a memory pool for executing the media application processing task is allocated.

[0106] The preset processing content duration can be set according to actual conditions. Specifically, the preset processing content duration can be automatically set according to the video GOP size and the machine memory configuration. For example, if the GOP size is 2 seconds and the machine memory size is sufficient, the preset processing content duration can be set to 2 seconds. The video GOP size is the time interval between two I frames in video encoding.

[0107] If the target media content is image content, the content attribute information may be image size of an image frame unit, image color sampling information, image pixel bit depth, and frame refresh frequency.

[0108] Image pixel bit depth refers to the bit depth required for each pixel when recording the color of a digital image. The computer can display colors because it uses a counting unit called "bit" to record the data of the color represented. When this data is recorded in the computer in a certain arrangement, it constitutes a computer file of a digital image. "Bit" is used to record the color value of each pixel. The richer the color of the image, the more "bits" there are. Specifically, N bits represent that a color contains 2 to the power of N different shades of color. The number of bits used for each pixel in the computer is the "bit depth".

[0109] Image color sampling information may refer to the sampling method of a YUV image, and the sampling methods may mainly include YUV 4:4:4 sampling, YUV 4:2:2 sampling, and YUV 4:2:0 sampling. Specifically, Y represents brightness (Luminance, Luma), while U and V represent chrominance (Chrominance, Chroma). Chrominance defines two aspects of color: hue and saturation. For a YUV image, not every pixel needs to contain the three components of Y, U, and V. Depending on the sampling method, each Y component may correspond to its own UV component, or several Y components may share the UV component. Regardless of the sampling method, the brightness is fully sampled, and the difference lies in the sampling rate of the U and V components.

[0110] Among them, in the YUV 4:4:4 sampling method, each Y corresponds to a set of UV components, and the sampling ratios of the three components of Y, U, and V are the same. Therefore, in the generated image, the three component information of each pixel is complete, all 8 bits, that is, one byte.

[0111] In the YUV 4:2:2 sampling method, every two Y components share a set of UV components, and the UV component is half of the Y component sampling. The Y component and UV component are sampled at a ratio of 2:1. If there are 10 pixels in the horizontal direction, 10 Y components are sampled, and only 5 UV components are sampled; specifically, it can be understood that the Y component is sampled for each pixel sampled, while the U and V components are sampled one at a time.

[0112] In the YUV 4:2:0 sampling method, every four Ys share a set of UV components, and the UV component is only one-fourth of the Y component. This does not mean that only the U component is sampled without the V component. Instead, it means that when scanning each line, only one chrominance component (U or V) is scanned and sampled with the Y component in a 2:1 manner. For example, when scanning the first line, YU is sampled in a 2:1 manner, and when scanning the second line, the YV component is sampled in a 2:1 manner. For each chrominance component, its horizontal and vertical sampling is 2:1 compared to the Y component.

[0113] Among them, if the target media content is audio content, the content attribute information can be the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit. The audio sampling bit depth (bit depth) refers to the number of bits used to represent the amplitude at a certain point in time; the more bits there are, the more accurate the amplitude changes that can be represented.

[0114] Optionally, in this embodiment, the target media content includes image content, and the content frame unit is an image frame unit of the image content;

[0115] The step of “allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content” may include:

[0116] Calculating the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel bit depth of the image frame unit;

[0117] Determining a memory application size according to the memory capacity and the frame refresh frequency corresponding to the image frame unit;

[0118] Based on the memory application size, a memory pool is allocated for executing a media application processing task on the image content.

[0119] The image size, image color sampling information and image pixel bit depth of the image frame unit can be fused to obtain the memory capacity occupied by the image frame unit; the fusion method can be multiplication. After obtaining the memory capacity occupied by the image frame unit, it can be multiplied by the frame refresh frequency of the image frame unit and the preset processing content duration to obtain the memory application size, which is the size of the memory pool to be applied.

[0120] For example, the image size of the image frame unit is 7680*4320, and the image pixel bit depth is 10 bits. In the case of an image pixel bit depth of 10 bits, two 2-bytes can be allocated to each pixel; in addition, the image color sampling information can be a YUV 4:2:2 sampling method. In the YUV 4:2:2 sampling method, the UV component is half of the Y component sampling. When the number of bytes occupied by Y is 1, the number of bytes occupied by UV is 0.5. Therefore, the image color sampling information can be 1.5 bytes, and the memory capacity occupied by the image frame unit can be 7680*4320*1.5*2*8bit=796,262,400bit; if the preset processing content duration is 2 seconds and the frame refresh frequency is 50FPS, the memory application size can be: 796,262,400bit*2*50=79,626,240,000bit.

[0121] Optionally, in this embodiment, the target media content includes audio content, and the content frame unit is an audio frame unit of the audio content;

[0122] The step of “allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content” may include:

[0123] Calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and the audio sampling bit depth corresponding to the audio frame unit;

[0124] Determine the memory application size according to the memory capacity and the preset processing content duration;

[0125] Based on the memory application size, a memory pool is allocated for executing a media application processing task on the audio content.

[0126] Among them, the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit can be fused to calculate the memory capacity occupied by the audio frame unit. Among them, the fusion method can be multiplication. After obtaining the memory capacity occupied by the audio frame unit, it can be multiplied by the preset processing content duration to reach the memory application size, and the memory application size is the size of the memory pool to be applied. The preset processing content duration can be set according to actual conditions, and this embodiment does not impose specific restrictions on this.

[0127] For example, if the audio sampling frequency corresponding to the audio frame unit is 8K and the audio sampling bit depth is 8 bits, the memory capacity occupied by the audio frame unit can be 8000*8bit=64000bit; if the preset processing content duration is 2 seconds, the memory application size can be 64000bit*2=128000bit.

[0128] 103. For each content frame unit, determine a target memory block for processing the content frame unit from the memory pool.

[0129] The memory blocks in the memory pool may be marked with usage status information, including an idle state and an occupied state; a target memory block may be selected from the memory blocks in the idle state. Specifically, a memory block corresponding to the content attribute information (such as audio sampling frequency or video resolution) of the content frame unit to be processed may also be selected.

[0130] Optionally, in some embodiments, before the step of “determining a target memory block for processing the content frame unit from the memory pool”, the following steps may also be included:

[0131] Set the preset free memory queue;

[0132] Storing the memory addresses of the memory blocks in the memory pool that are in an idle state in the preset idle memory queue;

[0133] The step of “determining a target memory block for processing the content frame unit from the memory pool” may include:

[0134] Taking out the target memory address from the preset free memory queue;

[0135] A memory block corresponding to the target memory address is determined as a target memory block for processing the content frame unit.

[0136] 104. Execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block.

[0137] 105. When the content processing subtask is completed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

[0138] Optionally, in this embodiment, the step of “executing the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block” may include:

[0139] Based on the preset execution order, determining a target content processing subtask to be processed currently from the media application processing tasks;

[0140] Executing the task processing logic of the target content processing subtask on the content frame unit through the target memory block to obtain a processed content frame unit;

[0141] The step of "when the content processing subtask is completed, based on the preset execution order, reusing the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and obtaining the target processed content frame unit after executing all the content processing subtasks in the media application processing task" may include:

[0142] When the target content processing subtask is completed, based on the preset execution order, a new target content processing subtask to be processed is determined from the media application processing tasks, and the processed content frame unit is used as a new content frame unit;

[0143] By reusing the target memory block, the step of executing the task processing logic of executing the target content processing subtask on the content frame unit is returned to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

[0144] Among them, each content processing subtask is executed serially and has a certain execution order. Through memory reuse, all content processing subtasks can be executed serially one by one to complete the media application processing task.

[0145] The new target content processing subtask may be the next content processing subtask of the original target content processing subtask, and the output of the previous content processing subtask is the input of the next content processing subtask.

[0146] The target processed content frame unit is obtained by executing all content processing subtasks on the content frame unit.

[0147] Optionally, in this embodiment, after the step of "executing the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit by reusing the target memory block based on the preset execution order to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task", the media content processing method may further include:

[0148] Determining an unprocessed content frame unit, wherein the unprocessed content frame unit is a content frame unit that does not execute the media application processing task;

[0149] By reusing the target memory block, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

[0150] Among them, this embodiment can also provide a second level of memory reuse. The first level of memory reuse refers to memory reuse during the processing of each content processing subtask for the same content frame unit. The second level of memory reuse means that after a content frame unit completes the media application processing task, the memory block occupied by it during the processing can be reused for other frames.

[0151] Optionally, in this embodiment, the step of “executing the task processing logic of each content processing subtask on the unprocessed content frame unit by reusing the target memory block to obtain a target processed content frame unit corresponding to the unprocessed content frame unit” may include:

[0152] Adding a target memory address corresponding to the target memory block to a preset free memory queue, wherein the preset free memory queue is used to store memory addresses of memory blocks in an idle state;

[0153] Taking the target memory address from the preset free memory queue;

[0154] The task processing logic of each content processing subtask is executed on the unprocessed content frame unit through the target memory block corresponding to the target memory address to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

[0155] After the media application processing task of a content frame unit is executed through the target memory block, the target memory block may be marked as being in an idle state and added to a preset idle memory queue.

[0156] Optionally, the preset free memory queue may be a FIFO (First In First Out) queue.

[0157] Specifically, in some embodiments, the memory blocks in the memory pool are marked with usage status information, and the usage status information includes an idle state and an occupied state;

[0158] After the step of "executing the task processing logic of each content processing subtask on the content frame unit based on the preset execution order by reusing the target memory block to obtain the target processed content frame unit", the following may also be included:

[0159] When it is detected that the target memory block is completely used by the content frame unit, updating the usage status information of the target memory block from an occupied state to an idle state;

[0160] Adding the target address information of the target memory block to a preset free memory queue;

[0161] Taking the target memory address from the preset free memory queue;

[0162] Through the memory block corresponding to the selected target memory address, based on the preset order, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain the target processed content frame unit, wherein the unprocessed content frame unit is a content frame unit that has not executed the media application processing task.

[0163] In a specific scenario, when the target media content is video image content, memory pools corresponding to multiple video resolutions can be preset, such as a memory pool corresponding to a video resolution of 8k, a memory pool corresponding to a resolution of 4k, etc. Each memory pool includes multiple memory blocks at corresponding video resolutions. If a certain video image content needs to be processed, the memory pool to be allocated can be determined based on the video resolution of the video image content. If there is no memory pool corresponding to the video resolution, memory can be applied for based on the image size, color sampling information, image pixel depth, and frame refresh frequency. The memory application size is M, and the memory pool applied for includes multiple memory blocks at the corresponding video resolution. The total size of all memory blocks is M.

[0164] In the process of applying filters to each image frame unit, if there is no temporal correlation between the image frame units, multiple frames can be processed in parallel through multiple memory blocks from the applied memory pool. It should be noted that the temporal correlation here means that for a frame that needs to refer to another frame for pixel resolution, this frame is temporally correlated with the referenced frame.

[0165] For each image frame unit, the filter processing flow includes super-resolution, color HDR, face, font, etc. The processing flow of each frame is to allocate a memory block for processing the frame, apply a certain filter to the frame (such as super-resolution), and then reuse the memory block for the next filter application (such as color HDR) until all filter applications are completed. The memory block can be marked as free and reused for other frames.

[0166] For example, specifically, the media application processing tasks of video images may include five content processing sub-tasks: super-resolution, color HDR (high dynamic range processing), face enhancement, text enhancement, and watermark removal. Each content processing sub-task corresponds to a filter application. The output of the filter application corresponding to super-resolution is the input of the filter application corresponding to color HDR, the output of the filter application corresponding to color HDR is the input of the filter application corresponding to face enhancement, the output of the filter application corresponding to face enhancement is the input of the filter application corresponding to text enhancement, and the output of the filter application corresponding to text enhancement is the input of the filter application corresponding to watermark removal. For each image frame unit to be processed, a target memory block for processing the image frame unit can be allocated according to its content attribute information; then, the task processing logic of super-resolution is performed on the image frame unit through the target memory block. After the content processing subtask of super-resolution is completed, the target memory block is not released, and the task processing logic of color HDR is performed on the image frame unit after super-resolution processing by reusing the target memory block; after the content processing subtask of color HDR is completed, the target memory block is not released, and the task processing logic of face enhancement is performed on the image frame unit after color HDR processing by reusing the target memory block; after the content processing subtask of face enhancement is completed, the target memory block is not released, and the task processing logic of text enhancement is performed on the image frame unit after face enhancement processing by reusing the target memory block; after the content processing subtask of text enhancement is completed, the target memory block is not released, and the task processing logic of de-watermarking is performed on the image frame unit after text enhancement processing by reusing the target memory block to obtain the de-watermarked image frame unit, which is the target processed image frame unit after all content processing subtasks are completed. Finally, if there are other image frame units that have not executed the media application processing task, the target memory block can be provided to these unprocessed image frame units for multiplexing, so as to execute the media application processing task on the unprocessed image frame units through the target memory block.

[0167] Among them, reference Figure 1d The specific process of performing super-resolution, color HDR, face enhancement, text enhancement, and watermark removal on a certain image frame unit by reusing the target memory block is described as follows:

[0168] 1. Write the image frame unit into the target memory block. The module corresponding to the content processing subtask "super-resolution" reads the image frame unit from the target memory block, uses the image frame unit as the input of the module corresponding to "super-resolution", and processes the image frame unit through the task processing logic of the module corresponding to "super-resolution". When the content processing subtask corresponding to "super-resolution" is completed, the image frame unit after super-resolution processing is obtained and output;

[0169] 2. Write the output super-resolution processed image frame unit into the target memory block. The module corresponding to the content processing subtask "color HDR" reads the super-resolution processed image frame unit from the target memory block, uses the super-resolution processed image frame unit as the input of the module corresponding to "color HDR", and processes the super-resolution processed image frame unit through the task processing logic of the module corresponding to "color HDR". When the content processing subtask corresponding to "color HDR" is completed, the image frame unit processed by color HDR is obtained and output;

[0170] 3. Write the output color HDR processed image frame unit into the target memory block. The module corresponding to the content processing subtask "face enhancement" reads the color HDR processed image frame unit from the target memory block, uses the color HDR processed image frame unit as the input of the module corresponding to "face enhancement", and processes the color HDR processed image frame unit through the task processing logic of the module corresponding to "face enhancement". When the content processing subtask corresponding to "face enhancement" is completed, the image frame unit processed by face enhancement is obtained and output;

[0171] 4. Write the output image frame unit after face enhancement processing into the target memory block. The module corresponding to the content processing subtask "text enhancement" reads the image frame unit after face enhancement processing from the target memory block, uses the image frame unit after face enhancement processing as the input of the module corresponding to "text enhancement", and processes the image frame unit after face enhancement processing through the task processing logic of the module corresponding to "text enhancement". When the content processing subtask corresponding to "text enhancement" is completed, the image frame unit after text enhancement processing is obtained and output;

[0172] 5. Write the output image frame unit after text enhancement processing into the target memory block. The module corresponding to the content processing subtask "de-watermarking" reads the image frame unit after text enhancement processing from the target memory block, uses the image frame unit after text enhancement processing as the input of the module corresponding to "de-watermarking", and processes the image frame unit after text enhancement processing through the task processing logic of the module corresponding to "de-watermarking". When the content processing subtask corresponding to "de-watermarking" is completed, the image frame unit after de-watermarking is obtained and output. The image frame unit after de-watermarking is the target processed image frame unit after all content processing subtasks are executed.

[0173] The audio processing is similar to the video processing.

[0174] In another specific scenario, when the target media content is audio content, multiple memory pools corresponding to audio sampling frequencies can be preset, such as a memory pool corresponding to an audio sampling frequency of 8k, a memory pool corresponding to an audio sampling frequency of 16k, etc. Each memory pool includes multiple memory blocks at corresponding audio sampling frequencies. If you need to process a certain audio content, you can determine which memory pool to allocate based on the audio sampling frequency of the audio content. If there is no memory pool corresponding to the audio sampling frequency, you can apply for memory based on the audio sampling frequency and audio sampling bit depth. The memory application size is N. The memory pool applied for includes multiple memory blocks at the corresponding audio sampling frequency, and the total size of all memory blocks is N.

[0175] In the process of applying filters to each audio frame unit, if there is no timing correlation between the audio frame units, multiple frames can be processed in parallel through multiple memory blocks from the applied memory pool. It should be noted that the timing correlation here means that for a frame that needs to refer to other frames before it can be parsed, this frame is time-correlated with the referenced frame.

[0176] For each audio frame unit, the filter processing flow includes denoising, de-burring, de-jittering, compression repair, etc. The processing flow of each frame is to allocate a memory block for processing the frame, apply a certain filter to the frame, and then reuse the memory block for the next filter application after the processing is completed. After all filter application processes are completed, the memory block can be marked as free and reused by other audio frame units.

[0177] For example, the media application processing task of audio may include four content processing sub-tasks: denoising, de-burring, de-jittering, and compression and repair. Each content processing sub-task corresponds to a filter application. The output of the filter application corresponding to denoising is the input of the filter application corresponding to de-burring, the output of the filter application corresponding to de-burring is the input of the filter application corresponding to de-jittering, and the output of the filter application corresponding to de-jittering is the input of the filter application corresponding to compression and repair. For each audio frame unit to be processed, a target memory block for processing the audio frame unit can be allocated according to its content attribute information; then, the task processing logic of denoising is performed on the audio frame unit through the target memory block. After the denoising content processing subtask is completed, the target memory block is not released, and the task processing logic of deburring is performed on the audio frame unit after the denoising process by reusing the target memory block; after the deburring content processing subtask is completed, the target memory block is not released, and the task processing logic of de-jittering is performed on the audio frame unit after the deburring process by reusing the target memory block; after the de-jitter content processing subtask is completed, the target memory block is not released, and the task processing logic of compression and repair is performed on the audio frame unit after the de-jitter process by reusing the target memory block to obtain the compressed and repaired audio frame unit, which is the target processed audio frame unit after all content processing subtasks are completed. Finally, if there are other audio frame units that have not executed the media application processing task, the target memory block can be provided to these unprocessed audio frame units for multiplexing, so as to execute the media application processing task on the unprocessed audio frame units through the target memory block.

[0178] Among them, reference Figure 1e The specific process of performing the four content processing subtasks of denoising, deburring, dejittering, and compression repair on a certain audio frame unit by reusing the target memory block is described as follows:

[0179] 1. Write the audio frame unit into the target memory block. The module corresponding to the content processing subtask "denoising" reads the audio frame unit from the target memory block, uses the audio frame unit as the input of the module corresponding to "denoising", and processes the audio frame unit through the task processing logic of the module corresponding to "denoising". When the content processing subtask corresponding to "denoising" is completed, the denoised audio frame unit is obtained and output;

[0180] 2. Write the output denoised audio frame unit into the target memory block. The module corresponding to the content processing subtask "deburring" reads the denoised audio frame unit from the target memory block, uses the denoised audio frame unit as the input of the module corresponding to "deburring", and processes the denoised audio frame unit through the task processing logic of the module corresponding to "deburring". When the content processing subtask corresponding to "deburring" is completed, the denoised audio frame unit is obtained and output;

[0181] 3. Write the output audio frame unit after de-burring processing into the target memory block. The module corresponding to the content processing subtask "de-jitter" reads the audio frame unit after de-burring processing from the target memory block, uses the audio frame unit after de-burring processing as the input of the module corresponding to "de-jitter", and processes the audio frame unit after de-burring processing through the task processing logic of the module corresponding to "de-jitter". When the content processing subtask corresponding to "de-jitter" is completed, the audio frame unit after de-jitter processing is obtained and output;

[0182] 4. Write the output de-jittered audio frame unit into the target memory block. The module corresponding to the content processing subtask "compression and repair" reads the de-jittered audio frame unit from the target memory block, uses the de-jittered audio frame unit as the input of the module corresponding to "compression and repair", and processes the de-jittered audio frame unit through the task processing logic of the module corresponding to "compression and repair". When the content processing subtask corresponding to "compression and repair" is completed, the compressed and repaired audio frame unit is obtained and output. The compressed and repaired audio frame unit is the target processed audio frame unit after all content processing subtasks are executed.

[0183] Optionally, in this embodiment, after the step of "based on the preset execution order, reusing the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and obtaining the target processed content frame unit after executing all content processing subtasks in the media application processing task", the media content processing method may further include:

[0184] After completing the media application processing tasks of all content frame units, the processed target media content is output, and the target memory blocks occupied during the processing are released back to the memory pool, wherein the processed target media content includes each target processed content frame unit.

[0185] Specifically, the output processed target media content can be used as the input of the content encoding module to be encoded, and then the encoded target media content is input to the mixed flow encapsulation module for container format encapsulation and compression, and then sent to the terminal.

[0186] In a specific scenario, Figure 1f As shown, it is a basic architecture diagram of the memory pool cache of the media content processing method provided by this application, and the basic processing flow is as follows:

[0187] The Filter processing flow of avfilter apply effects (application effect processing module) takes the output audio PCM and video YUV raw streams of the decoding module as input. When allocating memory for the processed audio and video streams, the Filter can obtain the memory pool of the corresponding gear memory space according to the audio sampling frequency and video resolution. It should be noted that the processing of audio and video is separate, and the memory and processing of the two are independent processes, and two memory pools will be applied for audio and video respectively.

[0188] Among them, the memory pool can be allocated according to the audio sampling frequency (such as 8K, 16K, 24K, 32K, 44.1K...), video resolution (such as 8K, 4K, 2K, 1080P, 720P, 560P and 8bit, 10bit, 12bit, 420 / 422 / 444 color sampling information...), if the memory pool does not have the audio sampling rate and video resolution corresponding to the input video source, the size of the memory pool to be allocated can be determined according to its specific content attribute information. Specifically, the bit position (bit depth) can be applied for memory according to (8bit, 10bit, 12bit) / 8+1)byte, and the memory pool and dynamic memory application are pre-applied according to the audio sampling rate, video resolution, bit depth and color sampling information. For example, for an application scenario where the audio sampling rate is 8K, the video adopts the YUV 4:2:2 sampling method, the image pixel depth is 10bit, and the frame refresh frequency is 50FPS, the memory pool size applied for audio and video according to 2 seconds (i.e., the preset processing content duration in the above embodiment) is as follows:

[0189] The size of the audio memory pool can be recorded as PCM Buffersize, PCM Buffersize = audio sampling rate * audio sampling bit depth * duration = 8000 * 8bit * 2 = 128000bit

[0190] The video memory pool size can be recorded as YUV Buffersize, YUV Buffersize = image size * color sampling information * image pixel depth * duration * frame refresh frequency = 7680*4320*1.5*(10 / 8+1)*8bit*2*50 = 79,626,240,000bit

[0191] Among them, 1.5 means that in the YUV 4:2:2 sampling mode, the UV component is half the Y component sampling. When the number of bytes occupied by Y is 1, the number of bytes occupied by UV is 0.5.

[0192] In the case of only one frame of audio or video, or multiple frames without temporal correlation between frames, filter application processing can be performed in frame parallel or gop granularity parallel. Each frame within the gop granularity can be temporally correlated.

[0193] For a certain content frame unit, the avfilter application can apply for a memory block from the memory pool as a frame buffer according to the decoded audio sampling rate and video resolution and frame rate. The avfilter application can reuse the buffer applied from the memory pool when processing in serial or parallel. The output of the previous Filter application is the input for the next Filter application. After the last Filter application is processed, the target processed content frame unit can be sent to the audio and video encoding kernel. The audio and video encoding kernel will reapply for memory and copy the output content of the last Filter due to its own different kernel memory layout architecture. After the audio and video encoding kernel completes the copying, it can release the cache applied from the memory pool to the memory pool for reuse, that is, reclaim the occupied memory into the memory pool.

[0194] The present application can provide a solution for improving media processing performance based on memory pool allocation. Specifically, the repeated memory application and release kernel calls during each Filter application processing in the existing framework can be improved by reusing the memory pool buffer. The solution provided by the present application can improve the engineering performance by more than 20% in an audio and video high sampling rate and high resolution application scenario (such as 8K, 44.1 7-channel application scenario).

[0195] As can be seen from the above, this embodiment can obtain target media content to be processed, wherein the target media content includes at least one content frame unit; according to the content attribute information of the target media content, a memory pool for executing a media application processing task is allocated, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, a target memory block for processing the content frame unit is determined from the memory pool; through the target memory block, the task processing logic of the content processing subtask in the media application processing task is executed on the content frame unit; when the content processing subtask is executed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain a target processed content frame unit after executing all the content processing subtasks in the media application processing task. This application can execute various content processing subtasks through memory reuse, and release the applied memory blocks back to the memory pool only after all media application processing tasks are finally completed. This can reduce the frequent memory allocation application and release between various content processing subtasks, thereby improving the media application processing performance.

[0196] According to the method described in the foregoing embodiment, the following will be further described in detail by taking the example of the media content processing device being specifically integrated in the server.

[0197] The present application embodiment provides a method for processing media content. Figure 2 As shown, the specific process of the media content processing method may be as follows:

[0198] 201. A server obtains target media content to be processed, where the target media content includes at least one content frame unit.

[0199] The target media content may be a video type of content, which may include image content and audio content. In some embodiments, the target media content may also include only audio content or image content, which is not specifically limited in this embodiment. The target media content to be processed may be media content that requires a media application processing task, and may specifically be media content that has been decapsulated and decoded.

[0200] A content frame unit may include only one content frame or multiple content frames, such as a content frame unit may include all frames in a gap (GOP) between two I frames (key frames). GOP refers to the time interval between two I frames in video encoding.

[0201] Specifically, if the target media content is image content, the content frame unit may be an image frame unit; if the target media content is audio content, the content frame unit may be an audio frame unit.

[0202] 202. The server obtains a preset mapping relationship set, where the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools.

[0203] The preset mapping relationship set may be a relationship table between preset content attributes and preset memory pools. Based on the content attribute information, the corresponding preset memory pool may be queried from the relationship table.

[0204] 203. The server searches the preset mapping relationship set for a preset memory pool corresponding to the content attribute information of the target media content.

[0205] The memory pool for executing the media application processing task can be applied for allocation according to the content attribute information of the target media content, where the content attribute information may be audio sampling frequency or video resolution, etc.

[0206] 204. The server allocates a memory pool for executing a media application processing task based on the queried preset memory pool, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0207] Among them, the media application processing task is the task of performing media processing on each content frame unit of the target media content. Media processing (MPS) can specifically refer to container format decapsulation, decoding, enhancement and application processing (such as superimposing watermarks, video screenshots, smart covers, smart editing, image quality enhancement, etc.), audio and video encoding format transcoding, packaging and distribution processing, etc.

[0208] In this embodiment, the media application processing task includes multiple links, each of which can be regarded as a content processing subtask. Different content processing subtasks can perform different types of content processing on content frame units. For example, the content processing subtask can be super-resolution processing, interpolation processing, or color enhancement processing.

[0209] Since both the application and release of memory are inefficient, the present application can apply for a larger block of memory (i.e., a memory pool, specifically, when the block of memory is insufficient, memory can be redistributed) at the beginning, and then, in the process of executing tasks, the corresponding memory resources can be taken out from this block of memory each time when needed. And after all the media application processing tasks are completed, the memory blocks occupied in the processing process are actually released to the operating system, which can reduce the number of memory application and release operations, thereby improving efficiency.

[0210] The memory pool may include at least one memory block corresponding to content attribute information, and the size of each memory block may correspond to the content attribute information of the target media content (such as audio sampling rate, video resolution, etc.). Specifically, the sizes of the memory blocks in the memory pool are the same, so that the size of the memory allocated each time is fixed, avoiding the possibility of memory fragmentation.

[0211] Optionally, in this embodiment, the media content processing method may further include:

[0212] When no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set, determining a memory application size based on the content attribute information of the target media content and a preset content processing duration;

[0213] Based on the memory application size, a memory pool for executing the media application processing task is allocated.

[0214] The preset processing content duration can be set according to actual conditions. Specifically, the preset processing content duration can be automatically set according to the video GOP size and the machine memory configuration. For example, if the GOP size is 2 seconds and the machine memory size is sufficient, the preset processing content duration can be set to 2 seconds. The video GOP size is the time interval between two I frames in video encoding.

[0215] If the target media content is image content, the content attribute information may be image size of an image frame unit, image color sampling information, image pixel depth, and frame refresh frequency.

[0216] Among them, if the target media content is audio content, the content attribute information can be the audio sampling frequency and audio sampling bit depth corresponding to the audio frame unit. The audio sampling bit depth (bit depth) refers to the number of bits used to represent the amplitude at a certain point in time; the more bits there are, the more accurate the amplitude changes that can be represented.

[0217] 205. The server determines, for each content frame unit, a target memory block from the memory pool for processing the content frame unit.

[0218] The memory blocks in the memory pool may be marked with usage status information, including an idle state and an occupied state; a target memory block may be selected from the memory blocks in the idle state. Specifically, a memory block corresponding to the content attribute information (such as audio sampling frequency or video resolution) of the content frame unit to be processed may also be selected.

[0219] Optionally, in some embodiments, before the step of “determining a target memory block for processing the content frame unit from the memory pool”, the following steps may also be included:

[0220] Set the preset free memory queue;

[0221] Storing the memory addresses of the memory blocks in the memory pool that are in an idle state in the preset idle memory queue;

[0222] The step of “determining a target memory block for processing the content frame unit from the memory pool” may include:

[0223] Taking out the target memory address from the preset free memory queue;

[0224] A memory block corresponding to the target memory address is determined as a target memory block for processing the content frame unit.

[0225] 206. The server executes the task processing logic of each content processing subtask on the content frame unit based on the preset execution order by reusing the target memory block to obtain a target processed content frame unit.

[0226] Optionally, in this embodiment, the step of “executing the task processing logic of each content processing subtask on the content frame unit based on the preset execution order by reusing the target memory block to obtain the target processed content frame unit” may include:

[0227] Based on the preset execution order, determining a target content processing subtask to be processed currently from the media application processing tasks;

[0228] Executing the task processing logic of the target content processing subtask on the content frame unit through the target memory block to obtain a processed content frame unit;

[0229] When the target content processing subtask is completed, based on the preset execution order, a new target content processing subtask to be processed is determined from the media application processing tasks, and the processed content frame unit is used as a new content frame unit;

[0230] By reusing the target memory block, returning to the step of executing the task processing logic of executing the target content processing subtask on the content frame unit, a target processed content frame unit is obtained.

[0231] Among them, each content processing subtask is executed serially and has a certain execution order. Through memory reuse, all content processing subtasks can be executed serially one by one to complete the media application processing task.

[0232] The new target content processing subtask may be the next content processing subtask of the original target content processing subtask, and the output of the previous content processing subtask is the input of the next content processing subtask.

[0233] The target processed content frame unit is obtained by executing all content processing subtasks on the content frame unit.

[0234] Optionally, in this embodiment, after the step of "executing the task processing logic of each content processing subtask on the content frame unit based on the preset execution order by reusing the target memory block to obtain the target processed content frame unit", the media content processing method may further include:

[0235] Determining an unprocessed content frame unit, wherein the unprocessed content frame unit is a content frame unit that does not execute the media application processing task;

[0236] By reusing the target memory block, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

[0237] Among them, this embodiment can also provide a second level of memory reuse. The first level of memory reuse refers to memory reuse during the processing of each content processing subtask for the same content frame unit. The second level of memory reuse means that after a content frame unit completes the media application processing task, the memory block occupied by it during the processing can be reused for other frames.

[0238] 207. After completing the media application processing tasks of all content frame units, the server outputs the processed target media content and releases the target memory blocks occupied during the processing back to the memory pool. The processed target media content includes each target processed content frame unit.

[0239] Specifically, the output processed target media content can be used as the input of the content encoding module to be encoded, and then the encoded target media content is input to the mixed flow encapsulation module for container format encapsulation and compression, and then sent to the terminal.

[0240] This application can propose a memory pool allocation solution, which reuses the memory pool when processing all filter applications behind the decoder decoding module, reducing the engineering performance consumption caused by frequent memory allocation applications and releases between filter applications, which can improve the engineering performance by 20%+ in high-resolution, high-FPS video filter processing application scenarios such as 4K and 8K. The solution provided by this application has a good application in video cloud media processing business products and is also well used in customers' 8K projects.

[0241] As can be seen from the above, in this embodiment, the target media content to be processed can be obtained through a server, and the target media content includes at least one content frame unit; a preset mapping relationship set is obtained, and the preset mapping relationship set includes a mapping relationship between a preset content attribute and a preset memory pool; a preset memory pool corresponding to the content attribute information of the target media content is queried in the preset mapping relationship set; based on the queried preset memory pool, a memory pool for executing a media application processing task is allocated, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, a target memory block for processing the content frame unit is determined from the memory pool; by reusing the target memory block, based on the preset execution order, the task processing logic of each content processing subtask is executed on the content frame unit to obtain a target processed content frame unit; after completing the media application processing tasks of all content frame units, the processed target media content is output, and the target memory blocks occupied during the processing are released back to the memory pool, and the processed target media content includes each target processed content frame unit.

[0242] This application can execute various content processing subtasks through memory reuse, and release the applied memory blocks back to the memory pool only after all media application processing tasks are finally completed. This can reduce the frequent memory allocation application and release between various content processing subtasks, thereby improving the media application processing performance.

[0243] In order to better implement the above method, the embodiment of the present application also provides a media content processing device, such as Figure 3 As shown, the media content processing device may include an acquisition unit 301, an allocation unit 302, a memory block determination unit 303, a first execution unit 304, and a second execution unit 305, as follows:

[0244] (1) Acquisition unit 301;

[0245] The acquisition unit is used to acquire target media content to be processed, where the target media content includes at least one content frame unit.

[0246] (2) allocation unit 302;

[0247] An allocation unit is used to allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information.

[0248] Optionally, in some embodiments of the present application, the allocation unit may include an acquisition subunit, a query subunit and an allocation subunit, as follows:

[0249] The acquisition subunit is used to acquire a preset mapping relationship set, wherein the preset mapping relationship set includes a mapping relationship between a preset content attribute and a preset memory pool;

[0250] A query subunit, configured to query a preset memory pool corresponding to the content attribute information of the target media content in the preset mapping relationship set;

[0251] The allocation subunit is used to allocate a memory pool for executing the media application processing task based on the queried preset memory pool.

[0252] Optionally, in some embodiments of the present application, the allocation unit may further include an application size determination subunit and a memory pool allocation subunit, as follows:

[0253] The application size determination subunit is used to determine the memory application size based on the content attribute information of the target media content and the preset processing content duration when no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set;

[0254] The memory pool allocation subunit is used to allocate a memory pool for executing the media application processing task based on the memory application size.

[0255] Optionally, in some embodiments of the present application, the target media content includes image content, the content frame unit is an image frame unit of the image content; the allocation unit may include a first calculation subunit, a first determination subunit and a first allocation subunit, as follows:

[0256] The first calculation subunit is used to calculate the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel depth of the image frame unit;

[0257] A first determining subunit, configured to determine a memory application size according to the memory capacity and a frame refresh frequency corresponding to the image frame unit;

[0258] The first allocation subunit is used to allocate a memory pool for executing a media application processing task on the image content based on the memory application size.

[0259] Optionally, in some embodiments of the present application, the target media content includes audio content, and the content frame unit is an audio frame unit of the audio content;

[0260] The allocation unit may include a second calculation subunit, a second determination subunit, and a second allocation subunit, as follows:

[0261] The second calculation subunit is used to calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and the audio sampling bit depth corresponding to the audio frame unit;

[0262] A second determining subunit is used to determine the memory application size according to the memory capacity and the preset processing content duration;

[0263] The second allocation subunit is used to allocate a memory pool for executing a media application processing task on the audio content based on the memory application size.

[0264] (3) memory block determination unit 303;

[0265] The memory block determination unit is used to determine, for each content frame unit, a target memory block for processing the content frame unit from the memory pool.

[0266] (4) a first execution unit 304;

[0267] The first execution unit is used to execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block.

[0268] (5) a second execution unit 305;

[0269] The second execution unit is used to reuse the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

[0270] Optionally, in some embodiments of the present application, the first execution unit may include a task determination subunit and an execution subunit, and the second execution unit may include a redetermination subunit and a return subunit, as follows:

[0271] The task determination sub-unit is used to determine the target content processing sub-task to be processed currently from the media application processing tasks based on the preset execution order;

[0272] An execution sub-unit, configured to execute the task processing logic of the target content processing sub-task on the content frame unit through the target memory block to obtain a processed content frame unit;

[0273] Re-determining a sub-unit, for determining a new current target content processing sub-task to be processed from the media application processing task based on the preset execution order when the target content processing sub-task is completed, and using the processed content frame unit as a new content frame unit;

[0274] The return subunit is used to return to the step of executing the task processing logic of executing the target content processing subtask on the content frame unit by reusing the target memory block, so as to obtain the target processed content frame unit after executing all content processing subtasks in the media application processing task.

[0275] Optionally, in some embodiments of the present application, the media content processing device may further include a multiplexing unit as follows:

[0276] The multiplexing unit is used to determine an unprocessed content frame unit, where the unprocessed content frame unit is a content frame unit that has not executed the media application processing task; by reusing the target memory block, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain a target processed content frame unit corresponding to the unprocessed content frame unit.

[0277] Optionally, in some embodiments of the present application, the multiplexing unit may include an adding subunit, a removing subunit, and a task executing subunit, as follows:

[0278] The adding subunit is used to add the target memory address corresponding to the target memory block to a preset free memory queue, and the preset free memory queue is used to store the memory addresses of the memory blocks in the free state;

[0279] A fetching subunit, used for fetching the target memory address from the preset free memory queue;

[0280] The task execution subunit is used to execute the task processing logic of each content processing subtask on the unprocessed content frame unit through the target memory block corresponding to the target memory address, so as to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

[0281] Optionally, in some embodiments of the present application, the media content processing device may further include an output unit as follows:

[0282] The output unit is used to output the processed target media content after completing the media application processing tasks of all content frame units, and release the target memory blocks occupied during the processing back to the memory pool, wherein the processed target media content includes each target processed content frame unit.

[0283] As can be seen from the above, in this embodiment, the acquisition unit 301 can acquire the target media content to be processed, and the target media content includes at least one content frame unit; the allocation unit 302 allocates a memory pool for executing a media application processing task according to the content attribute information of the target media content, and the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; the memory block determination unit 303 determines the target memory block for processing the content frame unit from the memory pool for each content frame unit; the first execution unit 304 executes the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; and the second execution unit 305 reuses the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is executed, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task. The present application can execute various content processing subtasks through memory reuse, which can reduce frequent memory allocation applications and releases between various content processing subtasks, thereby improving media application processing performance.

[0284] The present application also provides an electronic device, such as Figure 4 As shown, it shows a schematic diagram of the structure of an electronic device involved in an embodiment of the present application, and the electronic device may be a terminal or a server, etc. Specifically:

[0285] The electronic device may include components such as a processor 401 with one or more processing cores, a memory 402 with one or more computer-readable storage media, a power supply 403, and an input unit 404. Those skilled in the art will appreciate that Figure 4 The electronic device structure shown in the figure does not constitute a limitation on the electronic device, and may include more or fewer components than shown in the figure, or combine certain components, or arrange the components differently.

[0286] The processor 401 is the control center of the electronic device, and uses various interfaces and lines to connect various parts of the entire electronic device. By running or executing software programs and / or modules stored in the memory 402, and calling data stored in the memory 402, the processor 401 performs various functions of the electronic device and processes data. Optionally, the processor 401 may include one or more processing cores; preferably, the processor 401 may integrate an application processor and a modem processor, wherein the application processor mainly processes the operating system, user interface, and application programs, and the modem processor mainly processes wireless communications. It is understandable that the above-mentioned modem processor may not be integrated into the processor 401.

[0287] The memory 402 can be used to store software programs and modules. The processor 401 executes various functional applications and data processing by running the software programs and modules stored in the memory 402. The memory 402 may mainly include a program storage area and a data storage area, wherein the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc.; the data storage area may store data created according to the use of the electronic device, etc. In addition, the memory 402 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, or other volatile solid-state storage devices. Accordingly, the memory 402 may also include a memory controller to provide the processor 401 with access to the memory 402.

[0288] The electronic device also includes a power supply 403 for supplying power to each component. Preferably, the power supply 403 can be logically connected to the processor 401 through a power management system, so as to manage charging, discharging, power consumption and other functions through the power management system. The power supply 403 can also include one or more DC or AC power supplies, recharging systems, power failure detection circuits, power converters or inverters, power status indicators and other arbitrary components.

[0289] The electronic device may further include an input unit 404, which may be used to receive input digital or character information and generate keyboard, mouse, joystick, optical or trackball signal input related to user settings and function control.

[0290] Although not shown, the electronic device may further include a display unit, etc., which will not be described in detail herein. Specifically in this embodiment, the processor 401 in the electronic device will load the executable files corresponding to the processes of one or more application programs into the memory 402 according to the following instructions, and the processor 401 will run the application programs stored in the memory 402, thereby realizing various functions, as follows:

[0291] Acquire target media content to be processed, the target media content includes at least one content frame unit; allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, determine a target memory block for processing the content frame unit from the memory pool; execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; when the content processing subtask is completed, reuse the target memory block based on the preset execution order to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and obtain a target processed content frame unit after executing all content processing subtasks in the media application processing task.

[0292] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0293] As can be seen from the above, this embodiment can obtain the target media content to be processed, the target media content includes at least one content frame unit; according to the content attribute information of the target media content, a memory pool for executing a media application processing task is allocated, the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, a target memory block for processing the content frame unit is determined from the memory pool; through the target memory block, the task processing logic of the content processing subtask in the media application processing task is executed on the content frame unit; when the content processing subtask is executed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and the target processed content frame unit after executing all the content processing subtasks in the media application processing task is obtained. The present application can execute each content processing subtask through memory reuse, which can reduce the frequent memory allocation application and release between each content processing subtask, thereby improving the media application processing performance.

[0294] A person of ordinary skill in the art will appreciate that all or part of the steps in the various methods of the above embodiments may be completed by instructions, or by controlling related hardware through instructions. The instructions may be stored in a computer-readable storage medium and loaded and executed by a processor.

[0295] To this end, an embodiment of the present application provides a computer-readable storage medium, in which a plurality of instructions are stored, and the instructions can be loaded by a processor to execute the steps in any one of the media content processing methods provided in the embodiments of the present application. For example, the instructions can execute the following steps:

[0296] Acquire target media content to be processed, the target media content includes at least one content frame unit; allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; for each content frame unit, determine a target memory block for processing the content frame unit from the memory pool; execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; when the content processing subtask is completed, reuse the target memory block based on the preset execution order to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, and obtain a target processed content frame unit after executing all content processing subtasks in the media application processing task.

[0297] The specific implementation of the above operations can be found in the previous embodiments, which will not be described in detail here.

[0298] The computer-readable storage medium may include: a read-only memory (ROM), a random access memory (RAM), a magnetic disk or an optical disk, etc.

[0299] Since the instructions stored in the computer-readable storage medium can execute the steps in any media content processing method provided in the embodiments of the present application, the beneficial effects that can be achieved by any media content processing method provided in the embodiments of the present application can be achieved. Please refer to the previous embodiments for details and will not be repeated here.

[0300] According to one aspect of the present application, a computer program product or a computer program is provided, the computer program product or the computer program comprising computer instructions, the computer instructions being stored in a computer-readable storage medium. A processor of a computer device reads the computer instructions from the computer-readable storage medium, and the processor executes the computer instructions, so that the computer device executes the methods provided in various optional implementations of the above-mentioned media content processing aspects.

[0301] The above is a detailed introduction to a media content processing method and related equipment provided in an embodiment of the present application. Specific examples are used in this article to illustrate the principles and implementation methods of the present application. The description of the above embodiments is only used to help understand the method of the present application and its core idea. At the same time, for technical personnel in this field, according to the idea of ​​the present application, there will be changes in the specific implementation method and application scope. In summary, the content of this specification should not be understood as a limitation on the present application.

Claims

1. A media content processing method, characterized in that: include: Acquire target media content to be processed, wherein the target media content includes at least one content frame unit; Allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; For each content frame unit, determining a target memory block for processing the content frame unit from the memory pool; Executing the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; When the content processing subtask is completed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

2. The method according to claim 1, characterized in that The step of executing the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block includes: Based on the preset execution order, determining a target content processing subtask to be processed currently from the media application processing tasks; Executing the task processing logic of the target content processing subtask on the content frame unit through the target memory block to obtain a processed content frame unit; When the content processing subtask is completed, based on the preset execution order, the target memory block is reused to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task, including: When the target content processing subtask is completed, based on the preset execution order, a new target content processing subtask to be processed is determined from the media application processing tasks, and the processed content frame unit is used as a new content frame unit; By reusing the target memory block, the step of executing the task processing logic of executing the target content processing subtask on the content frame unit is returned to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

3. The method according to claim 1, characterized in that: The method further comprises: reusing the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order, and obtaining the target processed content frame unit after executing all the content processing subtasks in the media application processing task; Determining an unprocessed content frame unit, wherein the unprocessed content frame unit is a content frame unit that does not execute the media application processing task; By reusing the target memory block, the task processing logic of each content processing subtask is executed on the unprocessed content frame unit to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

4. The method according to claim 3, characterized in that The method of reusing the target memory block to execute the task processing logic of each content processing subtask on the unprocessed content frame unit to obtain the target processed content frame unit corresponding to the unprocessed content frame unit includes: Adding a target memory address corresponding to the target memory block to a preset free memory queue, wherein the preset free memory queue is used to store memory addresses of memory blocks in an idle state; Taking the target memory address from the preset free memory queue; The task processing logic of each content processing subtask is executed on the unprocessed content frame unit through the target memory block corresponding to the target memory address to obtain the target processed content frame unit corresponding to the unprocessed content frame unit.

5. The method according to claim 1, characterized in that The allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content includes: Obtaining a preset mapping relationship set, wherein the preset mapping relationship set includes mapping relationships between preset content attributes and preset memory pools; Querying a preset memory pool corresponding to the content attribute information of the target media content in the preset mapping relationship set; Based on the queried preset memory pool, a memory pool for executing the media application processing task is allocated.

6. The method according to claim 5, characterized in that The method further comprises: When no preset memory pool corresponding to the content attribute information can be found in the preset mapping relationship set, determining a memory application size based on the content attribute information of the target media content and a preset content processing duration; Based on the memory application size, a memory pool for executing the media application processing task is allocated.

7. The method according to claim 1, characterized in that The target media content includes image content, and the content frame unit is an image frame unit of the image content; The allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content includes: Calculating the memory capacity occupied by the image frame unit according to the image size, image color sampling information and image pixel bit depth of the image frame unit; Determining a memory application size according to the memory capacity and the frame refresh frequency corresponding to the image frame unit; Based on the memory application size, a memory pool is allocated for executing a media application processing task on the image content.

8. The method according to claim 1, characterized in that: The target media content includes audio content, and the content frame unit is an audio frame unit of the audio content; The allocating a memory pool for executing a media application processing task according to the content attribute information of the target media content includes: Calculate the memory capacity occupied by the audio frame unit according to the audio sampling frequency and the audio sampling bit depth corresponding to the audio frame unit; Determine the memory application size according to the memory capacity and the preset processing content duration; Based on the memory application size, a memory pool is allocated for executing a media application processing task on the audio content.

9. The method according to claim 1, characterized in that: The method further comprises: reusing the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order, and obtaining the target processed content frame unit after executing all the content processing subtasks in the media application processing task; After completing the media application processing tasks of all content frame units, the processed target media content is output, and the target memory blocks occupied during the processing are released back to the memory pool, wherein the processed target media content includes each target processed content frame unit.

10. A media content processing device, characterized in that: include: An acquisition unit, configured to acquire target media content to be processed, wherein the target media content includes at least one content frame unit; an allocating unit, configured to allocate a memory pool for executing a media application processing task according to content attribute information of the target media content, wherein the media application processing task includes at least one content processing subtask arranged in a preset execution order, and the memory pool includes at least one memory block corresponding to the content attribute information; A memory block determination unit, configured to determine, for each content frame unit, a target memory block for processing the content frame unit from the memory pool; A first execution unit, configured to execute the task processing logic of the content processing subtask in the media application processing task on the content frame unit through the target memory block; The second execution unit is used to reuse the target memory block to execute the task processing logic of the new content processing subtask in the media application processing task on the processed content frame unit based on the preset execution order when the content processing subtask is completed, so as to obtain the target processed content frame unit after executing all the content processing subtasks in the media application processing task.

11. An electronic device, characterized in that: It comprises a memory and a processor; the memory stores an application program, and the processor is used to run the application program in the memory to execute the operations in the media content processing method according to any one of claims 1 to 9.

12. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a plurality of instructions, and the instructions are suitable for being loaded by a processor to execute the steps in the media content processing method according to any one of claims 1 to 9.

13. A computer program product comprising a computer program or instructions, characterized in that When the computer program or instruction is executed by a processor, the steps in the media content processing method according to any one of claims 1 to 9 are implemented.

Citation Information

Cited By

  • Media content processing method and related device

    EP4668115A1