Media frame coding method and device, equipment and medium
By dividing media frames into multiple segments for parallel encoding, the problem of low media frame encoding efficiency is solved, achieving more efficient encoding and reduced latency.
Patent Information
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- BEIJING KINGSOFT CLOUD NETWORK TECH CO LTD
- Filing Date
- 2024-11-15
- Publication Date
- 2026-05-15
AI Technical Summary
The low encoding efficiency of media frames in existing technologies leads to large transmission delays, which affects the transmission efficiency of media frames.
The media frame to be encoded is divided into multiple segments, and these segments are encoded in parallel by multiple threads. The number of threads is less than or equal to the number of segments, and finally the encoded media frame is synthesized.
It improves the encoding efficiency of media frames and reduces transmission latency.
Smart Images
Figure CN122053846A_ABST
Abstract
Description
Technical Field
[0001] This disclosure relates to the field of coding technology, and in particular to a media frame coding method, apparatus, device and medium. Background Technology
[0002] In related technologies, media streams need to be encoded before transmission. For example, in live streaming scenarios, media frames in the live video stream need to be encoded first, and then the encoded media frames are sent to the receiving end. The encoding efficiency of media frames affects their transmission efficiency; lower encoding efficiency results in greater transmission latency. Therefore, improving the encoding efficiency of media frames is a technical problem that needs to be solved. Summary of the Invention
[0003] In order to solve the above-mentioned technical problems, or at least partially solve the above-mentioned technical problems, this disclosure provides a media frame encoding method, apparatus, device and medium.
[0004] In a first aspect, embodiments of this disclosure provide a media frame encoding method, the method comprising:
[0005] Divide the media frame to be encoded into multiple segments;
[0006] The multiple slices are encoded in parallel using multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices.
[0007] Based on the encoded segments, a new encoded media frame is synthesized.
[0008] In some implementations, the step of encoding the multiple slices in parallel using multiple threads to obtain encoded slices includes:
[0009] For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
[0010] In some implementations, after encoding the multiple slices in parallel using multiple threads to obtain the encoded slices, the method further includes:
[0011] Data of stationary objects is extracted from the encoded segments and used as inter-frame reference data.
[0012] In some implementations, dividing the media frame to be encoded into multiple segments includes:
[0013] When the media frame is a P-frame or a B-frame, the inter-frame reference data is decoded to obtain the decoded inter-frame reference data;
[0014] The residual data of the media frame is obtained by removing the decoded inter-frame reference data from the data of the media frame.
[0015] The residual data is fragmented to obtain multiple fragments.
[0016] In some implementations, the step of synthesizing the encoded media frame based on the encoded fragments includes:
[0017] Based on the fragments of the encoded residual data and the inter-frame reference data, the encoded media frame is synthesized.
[0018] In a second aspect, embodiments of this disclosure provide a media frame encoding apparatus, comprising:
[0019] The segmentation module is used to divide the media frame to be encoded into multiple segments;
[0020] An encoding module is used to encode the multiple slices in parallel using multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices;
[0021] The synthesis module is used to synthesize encoded media frames based on the encoded segments.
[0022] In some implementations, the encoding module is used for:
[0023] For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
[0024] In some embodiments, the apparatus further includes:
[0025] The extraction module is used to extract data of stationary objects from the encoded segments as inter-frame reference data.
[0026] In some implementations, the sharding module is used for:
[0027] When the media frame is a P-frame or a B-frame, the inter-frame reference data is decoded to obtain the decoded inter-frame reference data;
[0028] The residual data of the media frame is obtained by removing the decoded inter-frame reference data from the data of the media frame.
[0029] The residual data is fragmented to obtain multiple fragments.
[0030] In some implementations, the synthesis module is used for:
[0031] Based on the fragments of the encoded residual data and the inter-frame reference data, the encoded media frame is synthesized.
[0032] Thirdly, embodiments of this disclosure provide a computer device, including:
[0033] Memory;
[0034] Processor; and
[0035] Computer programs;
[0036] The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in the first aspect.
[0037] Fourthly, embodiments of this disclosure provide a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in the first aspect.
[0038] The media frame encoding method, apparatus, device, and medium provided in this disclosure divide the media frame to be encoded into multiple segments; encodes the multiple segments in parallel using multiple threads to obtain encoded segments, wherein the number of threads is less than or equal to the number of segments; and then synthesizes an encoded media frame based on the encoded segments. This disclosure improves the encoding efficiency and reduces latency by dividing the media frame to be encoded into multiple segments and using multiple threads to encode multiple segments of the same media frame in parallel. Attached Figure Description
[0039] The accompanying drawings, which are incorporated in and form a part of this specification, illustrate embodiments consistent with this disclosure and, together with the description, serve to explain the principles of this disclosure.
[0040] To more clearly illustrate the technical solutions in the embodiments of this disclosure or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.
[0041] Figure 1 This is a schematic diagram of a media frame encoding method provided by related technologies;
[0042] Figure 2 This is a flowchart of a media frame encoding method provided in an embodiment of this disclosure;
[0043] Figure 3 This is a schematic diagram of an encoding method provided in an embodiment of this disclosure;
[0044] Figure 4 This is a flowchart of a fragmentation method provided in an embodiment of this disclosure;
[0045] Figure 5 This is a schematic diagram of yet another encoding method provided in an embodiment of this disclosure;
[0046] Figure 6 This is a schematic diagram of the structure of a media frame encoding device provided in an embodiment of this disclosure;
[0047] Figure 7 This is a schematic diagram of the structure of a computer device embodiment provided in this disclosure. Detailed Implementation
[0048] To better understand the above-mentioned objectives, features, and advantages of this disclosure, the solutions disclosed herein will be further described below. It should be noted that, unless otherwise specified, the embodiments and features described herein can be combined with each other.
[0049] Numerous specific details are set forth in the following description in order to provide a full understanding of this disclosure, but this disclosure may also be implemented in other ways different from those described herein; obviously, the embodiments in the specification are only some, and not all, of the embodiments of this disclosure.
[0050] In the encoding methods provided by related technologies, each thread processes the encoding operation of one media frame (such as a video frame or audio frame) independently. Furthermore, P-frames must wait for I-frames to finish encoding before they can be encoded, and B-frames must wait for the previous or subsequent I-frame or P-frame to finish encoding before they can be encoded. Therefore, the encoding efficiency of a single media frame affects the overall encoding efficiency and latency. For example, Figure 1 This is a schematic diagram of a media frame encoding method provided by related technologies. Figure 1 Frame 1 can be exemplified as an I-frame, and thread 1 encodes it. Frame 2 can be exemplified as a P-frame, and thread 2 encodes it. Frame 3 can be exemplified as the next B-frame or P-frame, and thread 3 encodes it. During encoding, threads 1, 2, and 3 sequentially read frames 1, 2, and 3 from the task queue. After thread 1 completes encoding frame 1, thread 2 encodes frame 2 based on the inter-frame reference portion obtained by thread 1 (i.e., the encoded data of static objects contained in frame 1, also called inter-frame reference data). After thread 2 completes encoding, thread 3 encodes frame 3 based on the inter-frame reference data obtained by thread 2. The encoded frames 1, 2, and 3 are sorted, assembled, and sent to the receiving end via the main thread. It can be seen that thread 2 has at least a one-frame delay during encoding, while thread 3 has at least a two-frame delay, resulting in lower encoding efficiency and greater latency.
[0051] In view of the above-mentioned defects in related technologies, this disclosure provides a media frame encoding method, apparatus, device and medium to improve the encoding efficiency of a single media frame, thereby reducing latency as a whole.
[0052] The media frame encoding method, apparatus, device, and medium provided in this disclosure will be described below with reference to exemplary embodiments.
[0053] Example, Figure 2 This is a flowchart of a media frame encoding method provided in an embodiment of this disclosure. This method can be executed by a computer device, which can be understood as any device with computing and processing capabilities, such as a mobile phone, gateway, server, etc., but is not limited to the devices listed herein. Figure 2 As shown, in some embodiments, the media frame encoding method provided by this disclosure may include the following steps 201-203.
[0054] Step 201: Divide the media frame to be encoded into multiple segments.
[0055] In this embodiment of the disclosure, the media frame can be understood as a video frame or an audio frame.
[0056] In some implementations, "dividing the media frame to be encoded into multiple fragments" can be understood as dividing the media frame into fragments based on the number of threads used for encoding. For example, if there are n (n is a positive integer) threads used to encode the media frame, then the media frame can be divided into more than or equal to n fragments. Alternatively, in other implementations, the media frame can be divided into a corresponding number of fragments as needed, and the number of fragments can be less than the number of threads mentioned above. In the embodiments of this disclosure, each fragment contains a portion of the data of the media frame, and all fragments of the media frame are combined to obtain a complete media frame. Specific methods for dividing the media frame into multiple fragments can be found in related technologies and will not be elaborated here.
[0057] Step 202: Encode multiple shards in parallel using multiple threads to obtain encoded shards, wherein the number of threads is less than or equal to the number of shards.
[0058] In some implementations, each thread can encode one or more segments, and different threads can use the same encoding method to encode media frame segments. The specific encoding method can be set as needed, and this disclosure does not impose any specific limitations.
[0059] In some implementations, when a thread encodes a slice, it can use a preset hardware acceleration technique to encode the slice. For example, in one example, the thread can store the pixel blocks contained in the slice into a preset register, and then the central processing unit (CPU) reads the pixel blocks from the register and performs parallel encoding processing on the pixel blocks.
[0060] Taking I-frame as an example, Figure 3 This is a schematic diagram of an encoding method provided in an embodiment of this disclosure. Figure 3 In this example, the number of shards is equal to the number of threads. For example... Figure 3 As shown, in some implementations, an I-frame can be divided into multiple segments by a main thread, with each segment encoded by a separate thread. For any given thread, when encoding a segment, Single Instruction Multiple Data (SIMD) hardware acceleration technology can be used to encode the pixel blocks contained in the segment in parallel, resulting in encoded segments. The main thread then combines the segments encoded by multiple threads to synthesize the encoded video frame. Furthermore, after each thread completes encoding, it can extract data of stationary objects (such as trees, buildings, road signs, etc., but not limited to the objects listed here) from the encoded data and store them as inter-frame reference data.
[0061] When encoding P-frames or B-frames, one approach is to first fragment the P-frame or B-frame, then calculate the residual data for each fragment based on inter-frame reference data, and finally encode the residual data. The specific method for calculating the residual data based on inter-frame reference data can be found in relevant technologies and will not be elaborated here. It is understood that the above encoding method for P-frames or B-frames is merely an exemplary method and not the only method.
[0062] Step 203: Based on the encoded segments, synthesize the encoded media frames.
[0063] For example, in one exemplary manner, multiple encoded segments can be combined based on the position of each segment within the original media frame to obtain an encoded media frame. Alternatively, in other embodiments, the encoded segments can be packaged, with the package file including the position information of each segment within the media frame. Of course, the synthesis methods listed here are merely illustrative and not the only limitations. In fact, any method that can synthesize an encoded media frame based on encoded segments can be included within the protection scope of the embodiments of this disclosure.
[0064] This disclosure embodiment divides the media frame to be encoded into multiple segments; encodes the multiple segments in parallel using multiple threads to obtain encoded segments, wherein the number of threads is less than or equal to the number of segments; and then synthesizes an encoded media frame based on the encoded segments. This disclosure embodiment divides the media frame to be encoded into multiple segments and uses multiple threads to encode multiple segments of the same media frame in parallel, which can improve the encoding efficiency of the media frame and reduce latency.
[0065] Figure 4 This is a flowchart of a fragmentation method provided in an embodiment of this disclosure, such as... Figure 4 As shown, in some implementations, the media frame can be divided into multiple segments based on the method of steps 401-403, thereby accelerating the encoding efficiency by performing parallel encoding processing on multiple segments.
[0066] Step 401: When the media frame is a P-frame or a B-frame, decode the inter-frame reference data to obtain the decoded inter-frame reference data.
[0067] Step 402: Remove the decoded inter-frame reference data from the media frame data to obtain the residual data of the media frame.
[0068] Step 403: Perform fragmentation on the residual data to obtain multiple fragments.
[0069] The inter-frame reference data referred to in the embodiments of this disclosure can be extracted from previously encoded media frames (e.g., I-frames, P-frames, or B-frames) or fragments of media frames.
[0070] In this embodiment of the disclosure, inter-frame reference data can be understood as encoded data. The inter-frame reference data can be decoded by decoding the media frame or the segment of the media frame that has been encoded beforehand, so as to obtain the decoded inter-frame reference data.
[0071] In one implementation, when the media frame is a P-frame or a B-frame, the residual data of the media frame can be obtained by subtracting the decoded inter-frame reference data from the media frame data. This residual data can be understood, for example, as the data of moving objects (e.g., moving vehicles or people, but not limited to those listed here) contained in the media frame. Alternatively, in other ways, the data of moving objects can be identified from the media frame using a preset model, and this data can be used as the residual data.
[0072] When sharding residual data, the residual data can be divided into multiple shards as needed, or the residual data can be divided into multiple shards according to the number of threads used for encoding, so that the number of shards is greater than or equal to the number of threads. One thread processes one or more shards, and multiple threads encode multiple shards in parallel.
[0073] Furthermore, after obtaining the encoded segments, the encoded segments and inter-frame reference data can be combined according to the positions of each segment and inter-frame reference data in the media frame to obtain the encoded media frame.
[0074] for example, Figure 5 This is a schematic diagram of another encoding method provided in the embodiments of this disclosure, as shown below. Figure 5 As shown, when the media frame is a P-frame or B-frame, the residual data of the current media frame can be calculated based on the inter-frame reference data pre-stored in the preceding frame (i.e., the previously encoded media frame). Then, the main thread divides the residual data into multiple segments, each segment being encoded by a separate thread. The encoding of the segments can be accelerated using SIMD hardware acceleration technology, resulting in encoded segments. The main thread then combines the encoded segments with the inter-frame reference data to obtain the encoded media frame. Furthermore, the thread can extract data of still objects from the encoded segments and store it as inter-frame reference data.
[0075] By fragmenting the residual data of P-frames and B-frames and encoding multiple fragments in parallel using multiple threads, the encoding efficiency of P-frames and B-frames can be improved and latency reduced.
[0076] Figure 6 This is a schematic diagram of the structure of a media frame encoding device provided in an embodiment of this disclosure, as shown below. Figure 6 As shown, in one embodiment, the media frame encoding apparatus 60 provided in this disclosure includes:
[0077] Segmentation module 61 is used to divide the media frame to be encoded into multiple segments;
[0078] Encoding module 62 is used to encode the multiple slices in parallel based on multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices;
[0079] The synthesis module 63 is used to synthesize the encoded media frame based on the encoded fragments.
[0080] In some embodiments, the encoding module 62 is used for:
[0081] For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
[0082] In some embodiments, the apparatus further includes:
[0083] The extraction module is used to extract data of stationary objects from the encoded segments as inter-frame reference data.
[0084] In some implementations, the slicing module 61 is used for:
[0085] When the media frame is a P-frame or a B-frame, the inter-frame reference data is decoded to obtain the decoded inter-frame reference data;
[0086] The residual data of the media frame is obtained by removing the decoded inter-frame reference data from the data of the media frame.
[0087] The residual data is fragmented to obtain multiple fragments.
[0088] In some implementations, the synthesis module 63 is used for:
[0089] Based on the fragments of the encoded residual data and the inter-frame reference data, the encoded media frame is synthesized.
[0090] The apparatus provided in this disclosure is capable of executing the methods of any of the above method embodiments, and its execution mode and beneficial effects are similar, so they will not be described again here.
[0091] Figure 7 This is a schematic diagram of the structure of a computer device embodiment provided in this disclosure. (See attached diagram.) Figure 7 As shown, the computer device includes a memory 121 and a processor 122.
[0092] Memory 121 is used to store programs. In addition to programs, memory 121 can also be configured to store various other data to support operation on the computer device. Examples of such data include instructions for any application or method used on the computer device, images, videos, etc.
[0093] The memory 121 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk or optical disk.
[0094] The processor 122 is coupled to the memory 121 and executes the program stored in the memory 121 for:
[0095] Divide the media frame to be encoded into multiple segments;
[0096] The multiple slices are encoded in parallel using multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices.
[0097] Based on the encoded segments, a new encoded media frame is synthesized.
[0098] In some implementations, the step of encoding the multiple slices in parallel using multiple threads to obtain encoded slices includes:
[0099] For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
[0100] In some implementations, after encoding the multiple slices in parallel using multiple threads to obtain the encoded slices, the method further includes:
[0101] Data of stationary objects is extracted from the encoded segments and used as inter-frame reference data.
[0102] In some implementations, dividing the media frame to be encoded into multiple segments includes:
[0103] When the media frame is a P-frame or a B-frame, the inter-frame reference data is decoded to obtain the decoded inter-frame reference data;
[0104] The residual data of the media frame is obtained by removing the decoded inter-frame reference data from the data of the media frame.
[0105] The residual data is fragmented to obtain multiple fragments.
[0106] In some implementations, the step of synthesizing the encoded media frame based on the encoded fragments includes:
[0107] Based on the fragments of the encoded residual data and the inter-frame reference data, the encoded media frame is synthesized.
[0108] Furthermore, such as Figure 7 As shown, the computer device may also include other components such as a communication component 123, a power supply component 124, an audio component 125, and a display 126. Figure 7 The diagram only shows some components and does not mean that the computer device includes only these components. Figure 7 The components shown.
[0109] Communication component 123 is configured to facilitate wired or wireless communication between computer devices and other devices. The computer devices can access wireless networks based on communication standards, such as WiFi, 2G, or 3G, or combinations thereof. In one exemplary embodiment, communication component 123 receives broadcast signals or broadcast-related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, communication component 123 also includes a near-field communication (NFC) module to facilitate short-range communication. For example, the NFC module may be implemented based on radio frequency identification (RFID) technology, Infrared Data Association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology, and other technologies.
[0110] Power supply component 124 provides power to various components of the computer device. Power supply component 124 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power to the computer device.
[0111] Audio component 125 is configured to output and / or input audio signals. For example, audio component 125 includes a microphone (MIC) configured to receive external audio signals when the computer device is in an operating mode, such as call mode, recording mode, and voice recognition mode. The received audio signals may be further stored in memory 121 or transmitted via communication component 123. In some embodiments, audio component 125 also includes a speaker for outputting audio signals.
[0112] Display 126 includes a screen, which may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen can be implemented as a touchscreen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensors can sense not only the boundaries of the touch or swipe action but also the duration and pressure associated with the touch or swipe operation.
[0113] In addition, this disclosure also provides a computer-readable storage medium having a computer program stored thereon, the computer program being executed by a processor to implement the method described in any of the above method embodiments.
[0114] Exemplary embodiments of this disclosure also provide a computer program product, including a computer program, wherein, when executed by a computer's processor, the computer program is used to cause the computer to perform the method described in any of the above method embodiments.
[0115] It should be noted that, in this document, relational terms such as "first" and "second" are used merely to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.
[0116] The above description is merely a specific embodiment of this disclosure, enabling those skilled in the art to understand or implement it. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this disclosure. Therefore, this disclosure is not to be limited to the embodiments described herein, but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.
Claims
1. A media frame encoding method, characterized in that, The method includes: Divide the media frame to be encoded into multiple segments; The multiple slices are encoded in parallel using multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices. Based on the encoded segments, a new encoded media frame is synthesized.
2. The method according to claim 1, characterized in that, The process of encoding the multiple slices in parallel using multiple threads to obtain the encoded slices includes: For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
3. The method according to claim 1 or 2, characterized in that, After encoding the multiple slices in parallel using multiple threads to obtain the encoded slices, the method further includes: Data of stationary objects is extracted from the encoded segments and used as inter-frame reference data.
4. The method according to claim 1 or 2, characterized in that, The process of dividing the media frame to be encoded into multiple segments includes: When the media frame is a P-frame or a B-frame, the inter-frame reference data is decoded to obtain the decoded inter-frame reference data; The residual data of the media frame is obtained by removing the decoded inter-frame reference data from the data of the media frame. The residual data is fragmented to obtain multiple fragments.
5. The method according to claim 4, characterized in that, The process of synthesizing encoded media frames based on the encoded fragments includes: Based on the fragments of the encoded residual data and the inter-frame reference data, the encoded media frame is synthesized.
6. A media frame encoding apparatus, characterized in that, include: The segmentation module is used to divide the media frame to be encoded into multiple segments; An encoding module is used to encode the multiple slices in parallel using multiple threads to obtain encoded slices, wherein the number of threads is less than or equal to the number of slices; The synthesis module is used to synthesize encoded media frames based on the encoded segments.
7. The apparatus according to claim 6, characterized in that, The encoding module is used for: For each slice, the pixel blocks contained in the slice are encoded in parallel based on the thread to obtain the encoded slice.
8. The apparatus according to claim 6 or 7, characterized in that, The device further includes: The extraction module is used to extract data of stationary objects from the encoded segments as inter-frame reference data.
9. A computer device, wherein, include: Memory; processor; as well as Computer programs; The computer program is stored in the memory and configured to be executed by the processor to implement the method as described in any one of claims 1-5.
10. A computer-readable storage medium having a computer program stored thereon, wherein, When the computer program is executed by a processor, it implements the method as described in any one of claims 1-5.