Image data processing method and transcoding device

Abstract

The embodiment of the present application provides an image data processing method and a transcoding device, which relate to the field of data processing technology. The method is applied to a transcoding device, including: writing the image data of each frame into a first buffer in sequence according to the frame sequence; for each first buffer, decoding the image data of the current frame in the first buffer and writing it into a second buffer, setting the first buffer to an idle state and detecting whether there is a second buffer that meets the space release condition. If so, setting the second buffer that meets the space release condition to an idle state; for each second buffer, performing encoding preprocessing on the decoded data stored in the second buffer; encoding the frame structure data stored in each third buffer to obtain the transcoded image data of the target code stream; this solution can improve the efficiency of the code stream transcoding process.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image data processing method and a transcoding device. Background Art

[0002] In the field of image processing, before the bitstream is connected to the target device such as storage device, display device for processing, there is usually a need to transcode the bitstream to meet the target device's access requirements for resolution, frame rate, bit rate, etc. The so-called transcoding is to first decode the bitstream, and then encode the decoded data of each frame obtained by decoding to obtain a bitstream with a custom format or parameters (resolution, frame rate, and bit rate), that is, to obtain a bitstream that meets the access requirements of the target device.

[0003] In the related art, when transcoding a code stream, decoded data of each decoded frame is stored in a buffer, and the decoded data of each frame in the buffer is encoded by an encoder.

[0004] However, the related art processes the code stream in a serial manner, such as: a frame of image data is decoded and stored in a buffer, the encoder reads the decoded data of the frame from the buffer for encoding, and then the buffer is released and becomes idle, and then the image data of the next frame is decoded and stored in the buffer, and so on, which is inefficient.

[0005] It can be seen that in the related art, the efficiency of the code stream transcoding process is low. Summary of the invention

[0006] The purpose of the embodiment of the present application is to provide an image data processing method and a transcoding device to improve the efficiency of the transcoding process of the code stream. The specific technical solution is as follows:

[0007] In a first aspect, the present application provides an image data processing method, applied to a transcoding device, the method comprising:

[0008] Writing the image data of each frame in the target code stream into a first buffer in an idle state in the first buffer pool in sequence according to the frame order; wherein different first buffers store image data of different frames, and each first buffer is used to store image data of one frame;

[0009] For each first buffer storing image data, the image data of the current frame in the first buffer is decoded; the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool, the first buffer is set to an idle state, and it is detected whether there is a second buffer that meets the space release condition. In response to detecting the existence, the second buffer that meets the space release condition is set to an idle state; wherein different second buffers store decoded data of different frames, and each second buffer is used to store decoded data of one frame; the space release condition includes storing decoded data of a target frame located before the current frame and meeting the data release condition for the encoding stage of the transcoding process, and the target frame is a frame that is not referenced when decoding as a frame after the current frame;

[0010] For each second buffer storing decoded data, encoding preprocessing is performed on the decoded data stored in the second buffer, so that the decoded data stored in the second buffer is stored in a third buffer in a storage form of a frame structure and in a specified data format, thereby obtaining a third buffer storing frame structure data of one frame of the target code stream; wherein the specified data format is a data format required by the encoding stage of the transcoding process, different third buffers store frame structure data of different frames, and each third buffer is used to store frame structure data of one frame;

[0011] The frame structure data stored in each third buffer is encoded to obtain transcoded image data of the target code stream.

[0012] In a second aspect, the present application provides a transcoding device, including:

[0013] Memory, used to store computer programs;

[0014] The processor is used to implement any of the above-mentioned image data processing methods when executing the program stored in the memory.

[0015] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image data processing methods.

[0016] Beneficial effects of the embodiments of the present application:

[0017] The image data processing method provided in the embodiment of the present application is applied to a transcoding device. The present application processes the code stream in a parallel manner. The image data of each frame in the target code stream can be first written into a first buffer in an idle state in a first buffer pool in sequence according to the frame order; for each first buffer storing one frame of image data, the image data of the current frame in the first buffer is decoded, and the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool. In order to improve the efficiency of transcoding, the present application can set the first buffer to an idle state and detect whether there is a second buffer that meets the space release condition at present. In response to detecting There is a second buffer that meets the space release condition and is set to an idle state, so that the first buffer and the second buffer can be released in time to improve the efficiency of subsequent transcoding; for each second buffer storing a frame of decoded data, the decoded data stored in the second buffer can be encoded preprocessed to store the decoded data stored in the second buffer in a third buffer in the storage form of the frame structure and in the specified data format required for the encoding stage, so as to obtain a third buffer storing the frame structure data of a frame of the target code stream for subsequent encoding; thereafter, the frame structure data stored in each third buffer can be encoded to obtain the transcoded image data of the target code stream. The present application can transcode the code stream in a parallel manner through each first buffer, each second buffer and each third buffer, as well as the release of the buffer; it can be seen that the efficiency of the code stream transcoding process can be improved through this solution.

[0018] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0019] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0020] Figure 1 A schematic diagram of a flow chart of an image data processing method provided in an embodiment of the present application;

[0021] Figure 2 A schematic diagram of a decoding round-robin scheduling process provided in an embodiment of the present application;

[0022] Figure 3 A schematic diagram of a process for reducing frame rate encoding provided in an embodiment of the present application;

[0023] Figure 4A schematic diagram of a process for increasing frame rate encoding provided in an embodiment of the present application;

[0024] Figure 5 A schematic diagram of the process of coding rotation scheduling provided in an embodiment of the present application;

[0025] Figure 6 A schematic diagram of a transcoding device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0026] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0027] Related technologies can also control the processing rate of the post-stage VDP (video display processor) according to the preset timer time interval or the frame rate configured by the application layer, and then reversely block the processing rate of the VPSS (video processing sub system), and then reversely block the processing rate of the VDH (video decoding module for high definition, hardware video decoding engine). However, if the actual frame rate of the code stream is different from the frame rate configured by the application layer (for example, the actual frame rate is greater than the frame rate configured by the application layer), or when the code stream is played at a double speed, the actual frame rate may not be known, resulting in untimely frame transcoding processing and a large number of remaining frames, which leads to reverse blocking, abnormal code stream frame rate control, and low efficiency of the code stream transcoding process.

[0028] Based on this, the embodiment of the present application provides an image data processing method and a transcoding device to improve the efficiency of the transcoding process of the code stream; and the present application realizes the three functions of encoding, decoding and frame rate control through round-robin scheduling, and at the same time there is no anti-blocking mechanism, and the problems of anti-blocking and abnormal code stream frame rate control in related technologies will not occur.

[0029] The following first introduces an image data processing method provided by the present application.

[0030] Among them, the image data processing method provided by the present application can be applied to a transcoding device, which can be an electronic device with functions such as code stream decoding and encoding, such as a gateway device. The present application does not limit the specific form of the electronic device. In addition, the image data processing method provided by the embodiment of the present application can be applied to any scenario with code stream transcoding requirements. After transcoding the code stream, a code stream that meets the access requirements of the destination device can be obtained.

[0031] In addition, in the present application, multiple first buffers, multiple second buffers, and multiple third buffers may be provided, each first buffer is used to store one frame of image data, each second buffer is used to store one frame of decoded data, and each third buffer is used to store one frame of frame structure data, and different first buffers are used to store different frames of image data, different second buffers are used to store different frames of decoded data, and different third buffers are used to store different frames of frame structure data. By polling and releasing the stored data in each buffer, the image data of each frame of the code stream can be transcoded in parallel. Furthermore, each buffer may be a memory in a transcoding device, which is not limited in the present application.

[0032] Among them, an image data processing method provided in an embodiment of the present application is applied to a transcoding device, and the method includes:

[0033] Writing the image data of each frame in the target code stream into a first buffer in an idle state in the first buffer pool in sequence according to the frame order; wherein different first buffers store image data of different frames, and each first buffer is used to store image data of one frame;

[0034] For each first buffer storing image data, the image data of the current frame in the first buffer is decoded; the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool, the first buffer is set to an idle state, and it is detected whether there is a second buffer that meets the space release condition. In response to detecting the existence, the second buffer that meets the space release condition is set to an idle state; wherein different second buffers store decoded data of different frames, and each second buffer is used to store decoded data of one frame; the space release condition includes storing decoded data of a target frame located before the current frame and meeting the data release condition for the encoding stage of the transcoding process, and the target frame is a frame that is not referenced when decoding as a frame after the current frame;

[0035] For each second buffer storing decoded data, encoding preprocessing is performed on the decoded data stored in the second buffer, so that the decoded data stored in the second buffer is stored in a third buffer in a storage form of a frame structure and in a specified data format, thereby obtaining a third buffer storing frame structure data of one frame of the target code stream; wherein the specified data format is a data format required by the encoding stage of the transcoding process, different third buffers store frame structure data of different frames, and each third buffer is used to store frame structure data of one frame;

[0036] The frame structure data stored in each third buffer is encoded to obtain transcoded image data of the target code stream.

[0037] The image data processing method provided in the embodiment of the present application is applied to a transcoding device. The present application processes the code stream in a parallel manner. The image data of each frame in the target code stream can be first written into a first buffer in an idle state in a first buffer pool in sequence according to the frame order; for each first buffer storing one frame of image data, the image data of the current frame in the first buffer is decoded, and the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool. In order to improve the efficiency of transcoding, the present application can set the first buffer to an idle state and detect whether there is a second buffer that meets the space release condition at present. In response to detecting There is a second buffer that meets the space release condition and is set to an idle state, so that the first buffer and the second buffer can be released in time to improve the efficiency of subsequent transcoding; for each second buffer storing a frame of decoded data, the decoded data stored in the second buffer can be encoded preprocessed to store the decoded data stored in the second buffer in a third buffer in the storage form of the frame structure and in the specified data format required for the encoding stage, so as to obtain a third buffer storing the frame structure data of a frame of the target code stream for subsequent encoding; thereafter, the frame structure data stored in each third buffer can be encoded to obtain the transcoded image data of the target code stream. The present application can transcode the code stream in a parallel manner through each first buffer, each second buffer and each third buffer, as well as the release of the buffer; it can be seen that the efficiency of the code stream transcoding process can be improved through this solution.

[0038] An image data processing method provided by the present application is exemplarily introduced below in conjunction with the accompanying drawings.

[0039] like Figure 1 As shown, an image data processing method provided in an embodiment of the present application, applied to a transcoding device, may include the following steps:

[0040] S101: writing the image data of each frame in the target code stream into a first buffer in an idle state in a first buffer pool in sequence according to the frame sequence;

[0041] Wherein, different first buffers store image data of different frames, and each first buffer is used to store image data of one frame;

[0042] When transcoding the target code stream, the present application is different from the related art that can only process one frame of image data serially. The present application can process multiple frames of image data in parallel; first, the image data of each frame in the target code stream can be written into a first buffer in an idle state in the first buffer pool in sequence according to the frame sequence.

[0043] There are multiple first buffers in the first buffer pool, each of which is used to store image data of one frame, and different first buffers store image data of different frames, so that the image data of each frame can be decoded and processed in parallel later.

[0044] In addition, it should be noted that when the image data of each frame in the target code stream is sequentially written into a first buffer in the first buffer pool that is in an idle state according to the frame sequence, the first buffers for writing the image data of each frame in the target code stream may be first buffers that are adjacent in sequence, for example: before processing the target code stream, the first buffers are all in an idle state, at this time, the image data of the first frame is written into the first first buffer, the image data of the second frame is written into the second first buffer, and the image data of the Nth frame is written into the Nth first buffer; of course, before processing the target code stream, the transcoding device may process other code streams, at which time there may be occupied first buffers in the first buffer pool, so that the first buffers for writing the image data of each frame in the target code stream may be first buffers that are not in sequence and / or not necessarily adjacent, for example: the image data of the first frame is written into the third first buffer, the image data of the second frame is written into the fourth first buffer, the image data of the third frame is written into the ninth first buffer, and the image data of the fourth frame is written into the first first buffer (when writing the image data of the third frame, the first first buffer is occupied, and when writing the image data of the fourth frame, the first first buffer is in an idle state)...

[0045] That is, under the premise that the image data of each frame in the target code stream is sequentially written into a first buffer in an idle state in the first buffer pool according to the frame sequence, the present application does not limit the first buffer to be written.

[0046] It should be noted that when the image data of each frame in the target code stream is sequentially written to a first buffer in the first buffer pool that is in an idle state according to the frame sequence, due to the limited number of first buffers, the image data of some frames may not be simultaneously written to a first buffer in the first buffer pool that is in an idle state, and the image data of the remaining frames in the target code stream can be continued to be written in the frame sequence when there is a first buffer in the first buffer pool that is in an idle state. Of course, the number of first buffers may also be greater than the number of image data of each frame in the target code stream, so that after the image data of each frame of the target code stream is written, there may still be one or more first buffers in the first buffer pool that are in an idle state, which is reasonable.

[0047] S102: for each first buffer storing image data, decoding the image data of the current frame in the first buffer; writing the decoded data of the current frame obtained by decoding into a second buffer in the second buffer pool that is in an idle state, setting the first buffer to an idle state, and detecting whether there is a second buffer that meets the space release condition at present, and in response to detecting the existence of the second buffer, setting the second buffer that meets the space release condition to an idle state;

[0048] Wherein, different second buffers store decoded data of different frames, and each second buffer is used to store decoded data of one frame; the space release condition includes storing decoded data of a target frame located before the current frame and satisfying the data release condition of the encoding stage of the transcoding process, and the target frame is a frame that is not used as a reference when decoding a frame after the current frame;

[0049] For each first buffer storing image data, the image data of the current frame in the first buffer can be decoded, and the decoded data of the current frame obtained by decoding can be written into a second buffer in an idle state in the second buffer pool; illustratively, the image data of the current frame can be read from the first buffer and decoded, and the decoding process can be similar to the prior art, for example: the image data stored in each first buffer is allocated to the CPU according to the frame sequence, so that the CPU calls the decoder to decode the image data of each frame, etc., which will not be elaborated here, and the first buffer can be directly set to an idle state afterwards.

[0050] It is understandable that the decoding process of the image data of any frame may need to rely on the decoded data of other frames as a reference (the image data of each frame can be decoded and processed according to the frame sequence). After the decoded data of the current frame is stored in a second buffer in an idle state, the second buffer storing the decoded data of the current frame cannot be released immediately (because the current frame may be referenced when decoding subsequent frames, or the encoding stage of the transcoding process may occupy the second buffer, that is, considering the decoding stage and the encoding stage), it can be detected whether there is a second buffer that meets the space release condition, that is, the second buffer storing the decoded data of the video frame before the current frame is released (the decoded data of the video frame before the current frame may not be used as the frame referenced when decoding the frame after the current frame). The video frame before the current frame is determined according to the frame sequence, for example: the current frame is the fifth frame, and the video frames before the current frame are the first frame to the fourth frame; in response to detecting the existence, the second buffer that meets the space release condition is set to an idle state, thereby releasing the first buffer and the second buffer in time to improve the efficiency of code stream transcoding.

[0051] It should be noted that there are multiple second buffers in the second buffer pool, each second buffer is used to store the decoded data of one frame, and different second buffers store the decoded data of different frames. Due to the limited number of second buffers, there may be some frames of image data that have not been decoded and written to a second buffer in an idle state. When there is an idle second buffer in the second buffer pool, the image data of the remaining frames in the target code stream can be decoded in frame order, and the decoded data of one frame obtained can be written to a second buffer. Of course, the number of second buffers may also be greater than the number of decoded data of each frame in the target code stream, so that after writing the decoded data of each frame of the target code stream, there may still be one or more second buffers in an idle state in the second buffer pool, which is reasonable.

[0052] In addition, the spatial release conditions include storing decoded data of a target frame located before the current frame and satisfying the data release conditions for the encoding stage of the transcoding process, and the target frame is a frame that is not used as a reference when decoding the frame after the current frame; that is, in the present application, for each first buffer, the image data in the first buffer can be released directly after decoding; for each second buffer, since the decoding process of the image data of any frame may need to rely on the decoded data of other frames as a reference, or the encoding stage of the transcoding process may occupy the second buffer, therefore, for the second buffer storing the decoded data of any frame before the current frame, if the decoded data of the frame in the second buffer is not used as a reference when decoding the frame after the current frame, that is, the second buffer stores the decoded data of the target frame before the current frame; and, if the data release conditions for the encoding stage of the transcoding process are met, for example: the encoding stage does not occupy the second buffer, then the second buffer can be released and the second buffer becomes idle.

[0053] Exemplarily, the frame referenced when decoding any frame is an I frame, and / or a video frame before the frame that has a motion reference with the frame. It is understandable that the frame referenced when decoding any P frame (predictive-frame) is an I frame (intra-frame, I frame) before the frame. Of course, any P frame may refer to a video frame that has a motion reference with the P frame before the P frame to facilitate the decoding of the P frame. In addition, for different bitstream formats, the reference frames may be different. For example, in the H265 bitstream (the decoded data of the bitstream after decoding is in YUV (Y represents brightness, U and V represent chrominance and concentration respectively) format), the frame referenced when the current frame is decoded is the I frame, or the previous video frame of the frame that has a motion reference with the frame (for example, the current frame is the fourth frame, and the third frame may be referenced); in the smart (a video encoding and decoding technology that can be applied to network cameras, video recorders, etc.) format or SVAC (Scalable Video Coding, a video encoding standard) format (the decoded data before decoding is in smart format or SVAC format), the frame referenced when the current frame is decoded is the I frame, or the video frame that has a motion reference with the frame and has a fixed interval with the frame (for example, the current frame is the fourth frame, and the second frame may be referenced).

[0054] It should be emphasized that, for each first buffer storing image data, decoding processing can be performed if the image data of the current frame in the first buffer meets the decoding conditions; the so-called satisfying the decoding conditions can be understood as: the decoding data of other frames are not referenced when the current frame is decoded (for example, the current frame is an I frame), or, the decoding data of the frame required for reference when the current frame is decoded already exists (for example, the current frame is a P frame, and the frame required for reference when the current frame is decoded is the I frame before the current frame, and the decoding data of the I frame already exists; or, the current frame is a P frame, and the frame required for reference when the current frame is decoded is the I frame before the current frame and the previous P frame of the current frame, and the decoding data of the I frame before the current frame and the previous P frame of the current frame already exist).

[0055] S103: for each second buffer storing decoded data, performing encoding preprocessing on the decoded data stored in the second buffer, so as to store the decoded data stored in the second buffer in a third buffer in a storage form of a frame structure and in a specified data format, thereby obtaining a third buffer storing frame structure data of one frame of the target code stream;

[0056] Wherein, the specified data format is a data format required by the encoding stage of the transcoding process, different third buffers store frame structure data of different frames, and each third buffer is used to store the frame structure data of one frame;

[0057] For each second buffer storing decoded data, encoding preprocessing is performed on the decoded data stored in the second buffer, and the decoded data stored in the second buffer is stored in a third buffer in the storage form of the frame structure required for encoding and in a specified data format; of course, the third buffer is also an idle third buffer; and the present application can also be provided with a third buffer pool, which includes multiple third buffers. For each second buffer storing decoded data, an idle third buffer can be applied for from the third buffer pool so that the decoded data stored in the second buffer can be stored in the storage form of the frame structure and in a specified data format.

[0058] Exemplarily, in one implementation, the encoding preprocessing is performed on the decoded data stored in the second buffer so as to store the decoded data stored in the second buffer in a third buffer in a storage form of a frame structure and in a specified data format to obtain a third buffer storing frame structure data of one frame of the target code stream, including:

[0059] If the data format of the decoded data in the second buffer is different from the specified data format, the decoded data in the second buffer is converted into a format according to the specified data format, and the converted data is stored in a third buffer in an idle state in the third buffer pool in a frame structure form, so as to obtain a third buffer storing the frame structure data of one frame of the target code stream; wherein after the frame structure data of each third buffer in the third buffer pool is encoded, the third buffer is set to an idle state;

[0060] The method further comprises:

[0061] In response to the converted data being stored in a third buffer in an idle state in the third buffer pool in a frame structure, marking the second buffer as satisfying a data release condition for an encoding phase of a transcoding process;

[0062] Exemplarily, in another implementation, the encoding preprocessing is performed on the decoded data stored in the second buffer so as to store the decoded data stored in the second buffer in a third buffer in a storage form of a frame structure and in a specified data format to obtain a third buffer storing frame structure data of one frame of the target code stream, including:

[0063] If the data format of the decoded data in the second buffer is the same as the specified data format, determine the information about the frame structure of the decoded data in the second buffer, write the determined frame structure information and a pointer for indicating the address of the second buffer into the target memory area, use the target memory area as a third buffer, and obtain a third buffer storing the frame structure data of one frame of the target code stream; wherein the information about the frame structure is the structure information required to be used when the decoded data in the second buffer is encoded;

[0064] After the frame structure data in the obtained third buffer is encoded, the second buffer is marked as satisfying the data release condition for the encoding stage of the transcoding process.

[0065] In the present application, depending on whether the data format of the decoded data in the second buffer is the same as or different from the specified data format, different methods are used to store the decoded data stored in the second buffer in a third buffer according to the storage form of the frame structure and in the specified data format, thereby obtaining a third buffer storing the frame structure data of one frame of the target code stream.

[0066] If the data format of the decoded data in the second buffer is different from the specified data format, the decoded data in the second buffer needs to be format-converted according to the specified data format, and the converted data is stored in a third buffer in an idle state in the third buffer pool in the form of a frame structure, so as to obtain a third buffer storing the frame structure data of one frame of the target code stream; at this time, the second buffer is not occupied by the encoding stage of the transcoding process, and in response to the converted data being stored in a third buffer in an idle state in the third buffer pool in the form of a frame structure, the second buffer can be marked as satisfying the data release condition for the encoding stage of the transcoding process. Moreover, after the frame structure data of each third buffer in the third buffer pool is encoded, the third buffer can be set to an idle state to facilitate the storage of frame structure data of other frames, thereby improving the efficiency of the transcoding process.

[0067] If the data format of the decoded data in the second buffer is the same as the specified data format, the second buffer is mapped to a third buffer (i.e., in the encoding stage of the transcoding process, the second buffer is reused or occupied). Specifically, the information about the frame structure of the decoded data of the second buffer can be determined, and the determined frame structure information and the pointer used to indicate the address of the second buffer are written into the target memory area. At this time, the target memory area stores the frame structure information and the pointer indicating the second buffer (equivalent to the target memory area containing the second buffer and the decoded data stored in the second buffer). The target memory area can be used as a third buffer to obtain a third buffer storing the frame structure data of one frame of the target code stream; at this time, the second buffer is occupied by the encoding stage of the transcoding process. After the frame structure data of the second buffer in the obtained third buffer is encoded, the second buffer is marked as satisfying the data release condition for the encoding stage of the transcoding process, that is, the second buffer can be released to an idle state.

[0068] In addition, it should be emphasized that if the data format of the decoded data in the second buffer is the same as the specified data format, the decoded data in the second buffer can also be stored in a third buffer in an idle state in the third buffer pool in the form of a frame structure (no format conversion is required at this time), and this application does not limit this. Of course, if the data format of the decoded data in the second buffer is the same as the specified data format, by mapping the second buffer to a third buffer, the occupancy of the third buffer in the encoding stage of the transcoding process can be reduced, thereby improving the efficiency of the transcoding process. Moreover, when the data format of the decoded data in the second buffer is different from the specified data format, when the converted data is stored in a third buffer in an idle state in the third buffer pool in the form of a frame structure, information about the frame structure of the converted data can also be determined to adapt to the needs of the encoding stage.

[0069] Exemplarily, the information about the frame structure includes: information about the width and height of the frame, the format of the frame, the width and height span of the frame, frame coding and / or field coding. It should be noted that due to the different specified data formats required in the encoding stage, the required information about the frame structure can be flexibly adjusted, and the information about the frame structure can also include other information about the frame structure in the encoding stage. Under the condition of meeting the requirements of the encoding stage, this application does not limit the information about the frame structure.

[0070] Of course, regardless of whether it is necessary to map the second buffer to a third buffer, the method of storing the decoded data of the second buffer in the form of a frame structure can be to fill in the information about the frame structure and then store it; if the data format of the decoded data in the second buffer is different from the specified data format, that is, there is no need to map the second buffer to a third buffer, then it is necessary to first convert the decoded data in the second buffer into the specified data format, and then fill the converted data with information about the frame structure. This application does not limit this.

[0071] It should be noted that there are multiple third buffers in the third buffer pool, each of which is used to store the frame structure data of a frame, and different third buffers store the frame structure data of different frames. Due to the limited number of third buffers, there may be decoded data of some frames that have not been stored in a third buffer in an idle state in the form of a frame structure and in a specified data format. When there is a third buffer in an idle state in the third buffer pool, the decoded data of the remaining frames in the target code stream can be stored in the third buffer in an idle state in the form of a frame structure and in a specified data format according to the frame sequence. Of course, the number of third buffers may also be greater than the number of decoded data of each frame in the target code stream, so that after storing the frame structure data of each frame of the target code stream, there may still be one or more third buffers in an idle state in the third buffer pool, which is reasonable.

[0072] S104: Encoding the frame structure data stored in each third buffer to obtain transcoded image data of the target code stream.

[0073] After the frame structure data is stored in the third buffer, the frame structure data stored in each of the third buffers (each of the third buffers stores frame structure data of a different frame) may be encoded. Exemplarily, encoding the frame structure data stored in each of the third buffers to obtain transcoded image data of the target code stream includes:

[0074] The frame structure data stored in each third buffer is distributed to the target CPU according to the frame sequence, so that the target CPU calls the encoder to encode the received frame structure data to obtain the transcoded image data of the target code stream.

[0075] The present application can distribute the frame structure data stored in each third buffer to the target CPU according to the frame sequence, and the target CPU can call the soft encoder (for example: encoder program) or the hard encoder to encode the frame structure data of each received frame to obtain the transcoded image data of the target code stream. Among them, the way in which the target CPU calls the soft encoder or the hard encoder to encode the frame structure data of each received frame can be similar to the prior art, for example: first determine the encoding mode information (the encoding mode information can be calculated during the encoding process of the target code stream, or determined based on the encoded code stream (reference code stream) in a format different from the target code stream, or parsed in the decoding stage of the target code stream, or transmitted by the hardware encoder that encodes the reference code stream), and then encode the frame structure data of each frame according to the encoding mode information, which will not be elaborated here.

[0076] In the present application, frame rate control may also be performed before encoding, wherein the frame structure data stored in each third buffer is distributed to the target CPU according to the frame sequence, including:

[0077] Allocating the frame structure data stored in each third buffer to the target CPU according to the frame sequence and the processing method for frame rate control corresponding to the frame structure data of each frame;

[0078] The processing mode for frame rate control corresponding to the frame structure data of each frame is determined based on the value of the source frame rate, the value of the destination frame rate and the size relationship between the source frame rate and the destination frame rate, and the processing mode includes frame dropping processing, single frame sending processing or repeated frame sending processing; wherein, if the frame structure data of any frame is subjected to repeated frame sending processing, before the repeated frame sending processing is completed, the second buffer for storing the decoded data of the frame is not set to an idle state;

[0079] The source frame rate is the frame rate of the target code stream before transcoding, and the destination frame rate is the frame rate required by the code stream after transcoding.

[0080] When the frame structure data stored in each third buffer is distributed to the target CPU according to the frame sequence, it can be distributed to the target CPU according to the frame sequence and the processing method for frame rate control corresponding to the frame structure data of each frame; the processing method for frame rate control is used to control the frame rate of the target code stream, and can be determined based on the source frame rate before the target code stream is transcoded, the target frame rate required after the target code stream is transcoded (the frame rate can be manually configured or set to a default value, etc.), and the size relationship between the source frame rate and the target frame rate; adaptively, in order to achieve frame rate control, the processing method may include: frame drop processing, single frame sending processing or repeated frame sending processing, wherein the frame drop processing is used to reduce the frame rate (i.e., frame drop), the single frame sending processing is used to keep the frame rate unchanged, and the repeated frame sending processing is used to increase the frame rate (i.e., frame increase).

[0081] It should be noted that if the frame structure data of any frame is subjected to repeated frame sending processing, since the frame structure data of the frame may be stored in the third buffer obtained from the second buffer used to store the decoded data of the frame, in order to perform repeated frame sending processing on the frame, before the repeated frame sending processing of the frame is completed, the second buffer used to store the decoded data of the frame is not set to an idle state, that is, the second buffer storing the decoded data of the frame is not released, and the frame structure data of the frame can be repeatedly read from the third buffer obtained from the second buffer for repeated frame sending processing. Alternatively, the frame structure data of the frame is stored in a third buffer in the third buffer pool. After the frame structure data of each third buffer in the third buffer pool is encoded, the third buffer is set to an idle state. Therefore, in order to repeatedly send the frame, it is necessary to repeatedly obtain the frame structure data of the frame; that is, for each frame sending process except the first frame sending process, it is necessary to read the decoded data from the second buffer used to store the decoded data of the frame, and then generate the frame structure data of the frame and store it in a third buffer in the third buffer pool. Therefore, before the repeated frame sending process is completed, the second buffer used to store the decoded data of the frame is not set to an idle state.

[0082] Exemplarily, the process of determining the processing method for frame rate control corresponding to the frame structure data of each frame includes:

[0083] For the frame structure data of any frame, if the values ​​of the source frame rate and the destination frame rate are not less than the target value, the first control strategy is adopted to determine the processing method of the frame;

[0084] If the value of the source frame rate is not less than the target value, and the value of the destination frame rate is less than the target value, the second control strategy is adopted to determine the processing method of the frame;

[0085] If the value of the source frame rate is less than the target value, and the value of the destination frame rate is not less than the target value, the third control strategy is adopted to determine the processing method of the frame;

[0086] If the values ​​of the source frame rate and the destination frame rate are both less than the target value, the fourth control strategy is adopted to determine the processing method of the frame;

[0087] The first control strategy, the second control strategy, the third control strategy and the fourth control strategy are all control strategies for adapting the encoding frame rate in the encoding stage of the transcoding process to the target frame rate.

[0088] The frame rate can be divided according to the target value, and according to the relationship between the source frame rate, the target frame rate and the target value, and the size relationship between the original frame rate and the target frame rate, the first control strategy, the second control strategy, the third control strategy or the fourth control strategy that makes the encoding frame rate of the encoding stage of the transcoding process adapt to the target frame rate is adopted to determine the processing method for frame rate control corresponding to the frame structure data of each frame. In addition, the target value can be 1, and the target value is used to compare with the source frame rate and the target frame rate to determine the corresponding control strategy and determine the processing method of any frame.

[0089] Wherein, the first control strategy includes:

[0090] If the value of the source frame rate is greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the value of the destination frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the value of the source frame rate, and perform a separate frame sending process or a frame drop process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the value of the source frame rate, and update the parameter value of the specified parameter according to the calculated difference; the initial value of the specified parameter is 0;

[0091] If the value of the source frame rate is not greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the value of the source frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the value of the destination frame rate, and perform a separate frame sending process or a repeated frame sending process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the value of the destination frame rate, and update the parameter value of the specified parameter according to the calculated difference;

[0092] The second control strategy includes:

[0093] Calculate the sum of the current parameter value and the target value of the specified parameter, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the first value, and perform a separate frame sending process or a frame drop process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the first value, and update the parameter value of the specified parameter according to the calculated difference; the first value is the ratio of the value of the source frame rate to the value of the destination frame rate;

[0094] The third control strategy includes:

[0095] Calculate the sum of the current parameter value of the specified parameter and the second value, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the target value, and perform repeated frame sending processing on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is repeatedly sent, calculate the difference between the current parameter value of the specified parameter and the target value, and update the parameter value of the specified parameter according to the calculated difference; the second value is the ratio of the value of the target frame rate to the value of the source frame rate;

[0096] The fourth control strategy includes:

[0097] If the value of the source frame rate is greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and perform frame drop processing or separate frame sending processing on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to separate frame sending processing, calculate the difference between the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and update the parameter value of the specified parameter according to the calculated difference;

[0098] If the value of the source frame rate is not greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the relationship between the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and perform repeated frame sending or separate frame sending on the frame structure data of the frame according to the determination result; and when performing separate frame sending on the frame structure data of the frame, calculate the difference between the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and update the parameter value of the specified parameter according to the calculated difference.

[0099] In the present application, for each control strategy, according to the size relationship between the source frame rate value and the target frame rate value, the parameter value of the specified parameter is updated in different ways, and the size relationship between the current parameter value of the updated specified parameter and a value (the source frame rate value, the target frame rate value, the first numerical value, the target value, the inverse of the target frame rate value or the inverse of the source frame rate value) is determined, and according to the judgment result, the corresponding processing method is determined to be single frame sending processing, frame loss processing or repeated frame sending processing; and in a specific processing method, the parameter value of the specified parameter is updated in a corresponding manner, thereby realizing frame rate control in the transcoding process. Among them, for any frame, the specific process of determining the processing method of the frame according to the corresponding control strategy will be described in detail in the subsequent embodiments, which will not be repeated here.

[0100] In addition, the present application can also perform frame rate control in the stage before decoding, in the stage of obtaining decoded data after decoding, or in the stage after obtaining frame structure data but before encoding. The present application does not limit the timing of performing frame rate control.

[0101] It should be emphasized that the target CPU is at least one CPU allocated to the target code stream according to performance from among multiple CPUs.

[0102] In this application, N frames can be encoded in parallel at the same time to improve the efficiency of encoding, and different frames can be allocated to different CPUs for encoding. For example, a load balancing mode can be used to maximize the use of CPU resources. If there are two CPUs with the same performance in the transcoding device, the first frame is stored in the first third buffer, and the encoding of the first frame is run on CPU1; the second frame is stored in the second third buffer, and the encoding of the second frame is run on CPU2; the third frame is stored in the third third buffer, and the encoding of the third frame is run on CPU1, and so on.

[0103] When the performance of multiple CPUs in the transcoding device is different, multiple streams can be assigned to different CPUs for processing according to specific application scenarios. For example, if the processing performance of each of the 1st to 4th CPUs is 3Mbps, and the processing performance of each of the 5th to 8th CPUs is 1Mbps, then for multiple streams, in order to maximize the use of CPU performance, for example, a total of 8 streams, one solution is: first calculate the processing performance of the CPU required for each stream, traverse and calculate the sum of the single-channel processing performance value or the combined multi-channel processing performance value, and then compare it with 4Mbps (the processing performance of the last 4 CPUs (5th to 8th CPUs) is 1Mbps × 4), and get the optimal solution for multiple stream allocation.

[0104] For example: if the processing performance required for the 3rd code stream is less than or equal to 4Mbps and the value is closest to 4Mbps, the 3rd code stream is allocated to CPUs 5-8 for encoding in a load-balanced mode, and the other 7 code streams are allocated to CPUs 1-4 for encoding in a load-balanced mode. For another example: if the sum of the processing performance required for the 5th and 7th code streams is less than or equal to 4Mbps and the value is closest to 4Mbps, the 5th and 7th code streams are allocated to CPUs 5-8 for encoding in a load-balanced mode, and the other 6 code streams are allocated to CPUs 1-4 for encoding in a load-balanced mode.

[0105] It should be emphasized that the image data processing method provided by the present application can perform transcoding processing on one or more code streams. Any target code stream can be transcoded in the above-mentioned manner, and the target code streams can be transcoded in parallel according to the target CPU assigned to each target code stream, thereby improving the transcoding efficiency of the code stream.

[0106] In the technical solution of the present application, the operations of acquiring, storing, using, processing, transmitting, providing and disclosing the image data, decoding data, frame structure data, etc., are all carried out with the user's authorization.

[0107] The image data processing method provided in the embodiment of the present application is applied to a transcoding device. The present application processes the code stream in a parallel manner. The image data of each frame in the target code stream can be first written into a first buffer in an idle state in a first buffer pool in sequence according to the frame order; for each first buffer storing one frame of image data, the image data of the current frame in the first buffer is decoded, and the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool. In order to improve the efficiency of transcoding, the present application can set the first buffer to an idle state and detect whether there is a second buffer that meets the space release condition at present. In response to detecting There is a second buffer that meets the space release condition and is set to an idle state, so that the first buffer and the second buffer can be released in time to improve the efficiency of subsequent transcoding; for each second buffer storing a frame of decoded data, the decoded data stored in the second buffer can be encoded preprocessed to store the decoded data stored in the second buffer in a third buffer in the storage form of the frame structure and in the specified data format required for the encoding stage, so as to obtain a third buffer storing the frame structure data of a frame of the target code stream for subsequent encoding; thereafter, the frame structure data stored in each third buffer can be encoded to obtain the transcoded image data of the target code stream. The present application can transcode the code stream in a parallel manner through each first buffer, each second buffer and each third buffer, as well as the release of the buffer; it can be seen that the efficiency of the code stream transcoding process can be improved through this solution.

[0108] An image data processing method provided by the present application is described in detail below in conjunction with a specific embodiment.

[0109] The technical solution of the present application is a technical solution for transcoding round-robin scheduling, which uses software round-robin scheduling to implement decoding, frame rate control, and encoding mechanisms. It can be used in software-implemented round-robin scheduling solutions for decoding, frame rate control, and encoding (the decoding process and the encoding process can be implemented in software or hardware), thereby improving the processing efficiency of the transcoding process of the code stream.

[0110] The present application provides an image data processing method, which can realize transcoding functions including: decoding round-robin scheduling; frame rate control round-robin scheduling; encoding round-robin scheduling;

[0111] 1. Decoding round-robin scheduling;

[0112] In the related art, the buffer processing of the encoding process after decoding is serial processing, and the buffer can be released after use, but the speed of serial processing of the code stream is limited. In the decoding process of the present application, the threads can be increased through round-robin scheduling to parallelly process the image data of each frame of the same channel code stream to improve the decoding speed of the code stream (for example: each frame can call a decoding thread for decoding, and each frame is decoded in parallel. The decoding thread can be restored after decoding, or, for continuous code streams, multiple decoding threads can be continuously called to continuously decode each frame of the code stream); during the decoding process, the previous and next frames of the code stream have a reference dependency, so the code stream YUV buffer of the current frame (that is, the second buffer, which stores the decoded data of the current frame) is delayed to provide information for subsequent frame decoding (such as motion reference information, etc.); Figure 2 As shown, the decoding round-robin scheduling process is as follows:

[0113] Decoding input buffers 1~N in the decoding input buffer pool (i.e., multiple first buffers in the first buffer pool, which are all in an idle state by default) are polled and fed with the code stream data to be decoded (i.e., image data of each frame of the target code stream, each decoding input buffer is used to store image data of one frame, and different decoding input buffers store image data of different frames), and the decoded data is stored in code stream YUV buffers 1~N in the code stream YUV buffer pool (i.e., each second buffer in the second buffer pool) after decoding; for the image data of the current frame in any decoding input buffer, an idle code stream YUV buffer in the code stream YUV buffer pool is queried before decoding, and the decoded data after decoding the current frame is input into the idle code stream YUV buffer.

[0114] At the same time, it is detected whether there is any unreleased code stream YUV buffer for the decoded data of the frame before the current frame decoding process, that is, whether there is a second buffer that meets the space release condition. If not, continue to decode the image data of the next frame. Since there is still encoding processing in the later stage (the encoding stage may occupy the code stream YUV buffer of the current frame), the code stream YUV buffer of the current frame is not released for the time being. If it exists, continue to determine whether the decoded data of the frame in the unreleased code stream YUV buffer is a reference frame for subsequent frames (that is, the decoding of subsequent frames needs to obtain information from the frame in the unreleased code stream YUV buffer, such as: motion reference information), or whether the frame in the unreleased YUV buffer is occupied during the encoding process; when the frame in the unreleased code stream YUV buffer is no longer a reference frame and is no longer used in the encoding stage, the code stream YUV buffer is released, and then the code stream YUV buffer becomes idle; when the frame in the unreleased code stream YUV buffer is used as a reference frame for decoding subsequent frames, or the subsequent encoding process needs to occupy the code stream YUV buffer, the code stream YUV buffer is not released, and the decoding process of the image data of the next frame continues.

[0115] For each code stream YUV buffer storing decoded data, if the format of the decoded data is the same as the specified data format required by the encoding input, the code stream YUV buffer can be used as a code stream frame structure buffer (i.e., a third buffer in the third buffer pool) of the code stream frame structure buffer pool in the encoding stage. If the format of the decoded data is different from the specified data format required by the encoding input, YUV conversion is required before encoding (i.e., the decoded data is converted into a format with respect to the specified data format). The code stream frame structure buffers 1 to N used before encoding are also unified as a buffer pool. When decoding and outputting the code stream YUV buffer data of the current frame (i.e., the decoded data stored in the second buffer is stored in a third buffer in accordance with the storage form of the frame structure and in the specified data format), it is traversed to query whether there is an idle code stream frame structure buffer in the code stream frame structure buffer pool. If there is an idle code stream frame structure buffer, the idle code stream frame structure buffer is occupied as the current frame. The encoded code stream frame structure buffer stores the frame structure data after YUV conversion in the code stream frame structure buffer and sends it for encoding; after the encoding process is completed, the used code stream frame structure buffer is released (that is, after the frame structure data of each third buffer in the third buffer pool is encoded, the third buffer is set to an idle state); at this time, the encoding process will not affect the release of the code stream YUV buffer, but only affects the release of the code stream frame structure buffer. After the YUV conversion of the decoded data in the code stream YUV buffer of the current frame is completed, and it is subsequently determined that the current frame is not a frame referenced when decoding the frame after the current frame, the code stream YUV buffer can be released.

[0116] by Figure 2 Take the example of buffer zone occupation and release in each stage as follows:

[0117] 1. Before bitstream processing, all buffers of the decoding input buffer pool, bitstream YUV buffer pool, and bitstream frame structure buffer pool are in the released state, and the image data of each frame of the target bitstream is input into the decoding input buffer 1-N in sequence according to the frame order; the image data of the first frame is sent to the decoding input buffer 1, the image data of the second frame is sent to the decoding input buffer 2, and so on, and the image data of the Nth frame is sent to the decoding input buffer N.

[0118] 2. The YUV data (decoded data) of any frame of image data after decoding is placed in the code stream YUV buffer for storage. Assuming that the first frame is an I frame, the YUV data of the image data of the first frame after decoding is placed in the code stream YUV buffer 1. Since the subsequent code stream decoding needs to refer to the first frame I frame, after the YUV data of the first frame is YUV converted, when the code stream YUV buffer release detection is performed, the code stream YUV buffer 1 is not released, and the second frame processing continues.

[0119] 3. If the second frame is a P frame, the decoded YUV data of the image data of the second frame is put into the code stream YUV buffer 2. Since the decoding of the third frame needs to refer to the first frame I frame and the second frame P frame, after the YUV data of the second frame is YUV converted, when the code stream YUV buffer release detection is performed, neither the code stream YUV buffer 1 nor the code stream YUV buffer 2 is released, and the third frame processing continues.

[0120] 4. The third frame is a P frame, and the decoded YUV data of the image data of the third frame is placed in the code stream YUV buffer 3. Since the decoding of the fourth frame needs to refer to the first frame I frame and the third frame P frame, after the YUV data of the third frame is converted to YUV, the code stream YUV buffer 2 is released during the code stream YUV buffer release detection (at this time, it is not considered whether the encoding stage of the transcoding process occupies the code stream YUV buffer 2. It can be understood that the default encoding stage does not occupy the code stream YUV buffer 2; or, if the encoding stage occupies the code stream YUV buffer 2, the code stream YUV buffer 2 is not directly released, and the subsequent code stream YUV buffer release is similar), the code stream YUV buffer 1 and the code stream YUV buffer 3 are not released, and the fourth frame processing continues.

[0121] 5. The 4th frame is a P frame, and the YUV data of the decoded image data of the 4th frame is put into the code stream YUV buffer 4. Since the decoding of the 5th frame needs to refer to the 1st frame I frame and the 4th frame P frame, after the YUV data of the 4th frame is YUV converted, when the code stream YUV buffer release detection is performed, the code stream YUV buffer 3 is released, and the code stream YUV buffer 1 and the code stream YUV buffer 4 are not released, and the 5th frame processing continues.

[0122] 6. Continue to process the image data of each subsequent frame according to the above rules. For the bitstream in smart or SVAC format, the reference frame relationship is more complicated, and more frames need to be released later (for example: reference I frame and P frame at fixed intervals, such as: the 10th frame refers to the I frame of the first frame and the P frames of the 2nd, 4th, 6th and 8th frames).

[0123] 7. After using the code stream YUV buffer N, the processing of the next frame starts from the code stream YUV buffer 1 in the code stream YUV buffer pool in order. If the code stream YUV buffer 1 is still occupied and not released, the subsequent code stream YUV buffers will continue to be searched until the released code stream YUV buffer (that is, the idle code stream YUV buffer) is found, and the decoding process of the frame will continue.

[0124] 8. For the YUV data in any code stream YUV buffer, if the YUV data output by decoding does not meet the input requirements of the encoding (the data format of the decoded data in the second buffer is different from the specified data format), YUV conversion processing is performed before encoding (that is, the decoded data in the second buffer is format-converted according to the specified data format, and the converted data is stored in a third buffer in the third buffer pool in an idle state in the form of a frame structure). At this time, after the YUV data in the code stream YUV buffer is YUV-converted (the second buffer can be marked as satisfying the data release conditions for the encoding stage of the transcoding process), if the YUV data of the current frame in the code stream YUV buffer is not used as a frame for reference when decoding the frame after the current frame, it can be released immediately. Subsequent encoding processing will only limit the release of the code stream frame structure buffer in the code stream frame structure buffer pool.

[0125] 9. If the YUV data output by decoding meets the input requirements of the encoding (that is, the data format of the decoded data in the second buffer is the same as the specified data format), the code stream YUV buffer can be directly mapped as the code stream frame structure buffer for encoding input. At this time, the code stream YUV buffer cannot be released immediately, and it is necessary to detect whether the subsequent encoding process occupies the current code stream YUV buffer. If the encoding process is still using the current code stream YUV buffer, it cannot be released, and the processing of the next frame continues. If the encoding process no longer uses the current code stream YUV buffer, and the YUV data of the current frame in the code stream YUV buffer is not used as a reference frame for decoding the frame after the current frame, the current code stream YUV buffer is judged to be releasable and the buffer is released.

[0126] 2. Frame rate control in round-robin scheduling;

[0127] In the process of code stream transcoding in the related art, frame dropping processing can be selected (i.e., frame reduction of the code stream), but the buffer is released immediately after the current frame is processed, and the speed cannot be controlled to repeat frame sending processing, that is, the code stream cannot be increased. In the transcoding round-robin scheduling of the present application, the screening algorithm can be used to control whether the YUV data (decoded data) before encoding is sent for encoding, thereby controlling the frame rate of multiple source frame rates by frame rate reduction encoding and frame rate increase encoding. (That is, the frame structure data of each frame of the target code stream stored in each third buffer is distributed to the target CPU according to the frame sequence and the processing method for frame rate control corresponding to the frame structure data of each frame; the processing method for frame rate control corresponding to the frame structure data of each frame is determined based on the value of the source frame rate, the value of the target frame rate and the size relationship between the source frame rate and the target frame rate, and the processing method includes frame loss processing, single frame sending processing or repeated frame sending processing; wherein, if the frame structure data of any frame is subjected to repeated frame sending processing, before the repeated frame sending processing is completed, the second buffer for storing the decoded data of the frame is not set to an idle state; wherein, the source frame rate is the frame rate of the target code stream before transcoding, and the target frame rate is the frame rate required for the code stream after transcoding.)

[0128] This application performs frame rate control processing in the processing stage of the code stream frame structure buffer, such as Figure 3 As shown, the present application performs frame rate reduction encoding on the frame structure data in the code stream frame structure buffer 1-N. Specifically, the code stream frame structure buffer 1 and the corresponding reference code stream buffer 1, the code stream frame structure buffer 3 and the corresponding reference code stream buffer 3...the code stream frame structure buffer N and the corresponding reference code stream buffer N are sent to the encoding, and are output to the encoding output buffer 1, the encoding output buffer 2...the encoding output buffer N / 2 (N is an odd number, N / 2 can be rounded up) respectively; and, the code stream frame structure buffer 2 and the corresponding reference code stream buffer 2, the code stream frame structure buffer 4 and the corresponding reference code stream buffer 4...the code stream frame structure buffer N-1 and the corresponding reference code stream buffer N-1 are subjected to frame loss processing.

[0129] like Figure 4As shown, the present application performs frame rate increase encoding on the frame structure data in the code stream frame structure buffer 1-N. Specifically, the code stream frame structure buffer 1 and the corresponding reference code stream buffer 1, the code stream frame structure buffer 2 and the corresponding reference code stream buffer 2...the code stream frame structure buffer N and the corresponding reference code stream buffer N, repeatedly send frames for frame rate increase encoding, and output the encoded image data of the code stream frame structure buffer 1 and the corresponding reference code stream buffer 1 to the encoding output buffer 1 and the encoding output buffer 2, and output the encoded image data of the code stream frame structure buffer 2 and the corresponding reference code stream buffer 2 to the encoding output buffer 3 and the encoding output buffer 4, and output the encoded image data of the code stream frame structure buffer N and the corresponding reference code stream buffer N to the encoding output buffer 2N-1 and the encoding output buffer 2N. In addition, when performing frame rate up-coding or frame rate down-coding, the reference code stream buffer may not be used, and the frame structure data in the code stream frame structure buffer may be directly encoded; the reference code stream buffer stores the corresponding image frame of the corresponding reference code stream, and the image frame carries the encoding mode information; that is, the reference code stream buffer carries the encoding mode information, so that the frame structure data in the code stream frame structure buffer can be quickly encoded without recalculating the encoding mode information; of course, the encoding mode information can also be recalculated to directly encode the frame structure data in the code stream frame structure buffer, which is all reasonable.

[0130] The frame rate control processing method corresponding to the frame structure data of each frame is determined as follows:

[0131] 1. Source frame rate srcfps>=1, destination frame rate dstfps>=1,

[0132] (1) srcfps>dstfps, encode and output according to the destination frame rate dstfps, filter each frame in the code stream YUV buffer of the source frame rate srcfps to determine whether to send it to the subsequent processing. The filtering rule for each frame is: sumfps+=dstfps (i.e. sumfps= sumfps+dstfps), judge sumfps>=srcfps, if it is TRUE, the current frame is selected for subsequent processing, and sumfps-=srcfps (i.e. sumfps=sumfps-srcfps), if it is FALSE, the current frame is discarded. sumfps is the specified parameter, and the initial value is 0.

[0133] For example: the source frame rate srcfps is 25, the destination frame rate dstfps is 18,

[0134] At the first frame, sumfps+18=18, sumfps<25, the current frame is discarded (frame loss processing);

[0135] In the second frame, sumfps+18=36, sumfps>=25, the current frame is retained (sent to frame processing separately), and sumfps-25=11;

[0136] At the third frame, sumfps+18=29, sumfps>=25, the current frame is retained, and sumfps-25=4;

[0137] At the 4th frame, sumfps+18=22, sumfps<25, and the current frame is discarded;

[0138] At the 5th frame, sumfps+18=40, sumfps>=25, the current frame is retained, and sumfps-25=15;

[0139] At the 6th frame, sumfps+18=33, sumfps>=25, the current frame is retained, and sumfps-25=8;

[0140] At the 7th frame, sumfps+18=26, sumfps>=25, the current frame is retained, and sumfps-25=1;

[0141] At the 8th frame, sumfps+18=19, sumfps<25, and the current frame is discarded;

[0142] At the 9th frame, sumfps+18=37, sumfps>=25, the current frame is retained, and sumfps-25=12;

[0143] At the 10th frame, sumfps+18=30, sumfps>=25, the current frame is retained, and sumfps-25=5;

[0144] At the 11th frame, sumfps+18=23, sumfps<25, and the current frame is discarded;

[0145] At the 12th frame, sumfps+18=41, sumfps>=25, the current frame is retained, and sumfps-25=16;

[0146] At the 13th frame, sumfps+18=34, sumfps>=25, the current frame is retained, and sumfps-25=9;

[0147] At the 14th frame, sumfps+18=27, sumfps>=25, the current frame is retained, and sumfps-25=2;

[0148] At the 15th frame, sumfps+18=20, sumfps<25, and the current frame is discarded;

[0149] At the 16th frame, sumfps + 18 = 38, sumfps >= 25, the current frame is retained, and sumfps - 25 = 13;

[0150] At the 17th frame, sumfps + 18 = 31, sumfps >= 25, the current frame is retained, and sumfps - 25 = 6;

[0151] At the 18th frame, sumfps + 18 = 24, sumfps < 25, the current frame is discarded;

[0152] At the 19th frame, sumfps + 18 = 42, sumfps >= 25, the current frame is retained, and sumfps - 25 = 17;

[0153] At the 20th frame, sumfps + 18 = 35, sumfps >= 25, the current frame is retained, and sumfps - 25 = 10;

[0154] At the 21st frame, sumfps + 18 = 28, sumfps >= 25, the current frame is retained, and sumfps - 25 = 3;

[0155] At the 22nd frame, sumfps + 18 = 21, sumfps < 25, the current frame is discarded;

[0156] At the 23rd frame, sumfps + 18 = 39, sumfps >= 25, the current frame is retained, and sumfps - 25 = 14;

[0157] At the 24th frame, sumfps + 18 = 32, sumfps >= 25, the current frame is retained, and sumfps - 25 = 7;

[0158] At the 25th frame, sumfps + 18 = 25, sumfps >= 25, the current frame is retained, and sumfps - 25 = 0;

[0159] (2)srcfps <= dstfps. Optionally, it can be encoded and output according to the source frame rate srcfps, that is, the frame rate of the source code stream remains unchanged (for example: if the importance of this source code stream is relatively low (such as the importance coefficient is lower than the threshold), there is no need to consume resources to increase the frame). Optionally, it can be encoded and output according to the destination frame rate dstfps (for example: if the importance of this source code stream is relatively high (such as the importance coefficient is higher than the threshold), the source frame rate can be increased). It is necessary to determine whether each frame in the YUV buffer of the code stream in the source frame rate srcfps is repeatedly sent for encoding. The repetition rule for each frame is: sumfps += srcfps. If sumfps < dstfps, the current frame is repeatedly sent. If sumfps >= dstfps, the current frame is sent separately, and sumfps -= dstfps; and then continue to process the next frame.

[0160] For example: the source frame rate srcfps is 18, the destination frame rate dstfps is 25,

[0161] For the first frame, sumfps+18=18, sumfps<25, the current frame is sent repeatedly (repeated frame processing);

[0162] The first frame is repeatedly sent sumfps+18=36, sumfps>=25, the current frame is sent in (sent frame processing separately), and sumfps-25=11;

[0163] At the second frame, sumfps+18=29, sumfps>=25, the current frame is sent in, and sumfps-25=4;

[0164] 2. Source frame rate srcfps>=1, destination frame rate dstfps<1,

[0165] Encode and output according to the destination frame rate dstfps, such as dstfps=1 / N, and filter each frame in the code stream YUV buffer of the source frame rate srcfps to determine whether to send it to subsequent processing. The filtering rule for each frame is: sumfps+=1, judge sumfps>=srcfps×N, if it is TRUE, the current frame is selected for subsequent processing, and sumfps-=srcfps×N, if it is FALSE, the current frame is discarded.

[0166] For example: the source frame rate srcfps is 18, the destination frame rate dstfps is 1 / 4,

[0167] At the first frame, sumfps+1=1, sumfps<18×4, the current frame is discarded (frame loss processing);

[0168] At the second frame, sumfps+1=2, sumfps<18×4, the current frame is discarded (frame loss processing);

[0169] …

[0170] At the 72nd frame, sumfps+1=72, sumfps>=18×4, the current frame is retained (sent to frame processing separately), and sumfps-72=0;

[0171] 3. Source frame rate srcfps<1, destination frame rate dstfps>=1,

[0172] Optionally, the code can be output according to the source frame rate srcfps, that is, the source code stream frame rate is not changed. Another option is to encode and output according to the destination frame rate dstfps, such as srcfps=1 / M. It is necessary to determine whether each frame in the code stream YUV buffer in the source frame rate srcfps is resent for encoding. The repetition rule for each frame is: sumfps+=dstfps×M. If sumfps>=1 is judged, if it is TRUE, the current frame is repeated and sent to the subsequent processing, and sumfps-=1, if it is FALSE, then continue to process the next frame.

[0173] For example: the source frame rate srcfps is 1 / 4, the destination frame rate dstfps is 18,

[0174] For the first frame, sumfps+18×4=72, sumfps>=1, the current frame is sent repeatedly (repeated frame processing), and sumfps-1=71;

[0175] The first frame is sent repeatedly with sumfps=71, sumfps>=1, the current frame is sent repeatedly, and sumfps-1=70;

[0176] …

[0177] The first frame is repeatedly sent, sumfps=0, sumfps<1, and the current frame is no longer sent (that is, the repeated frame sending process is completed, and the frame is no longer repeatedly sent);

[0178] In the second frame, sumfps+18×4=72, sumfps>=1, the current frame is sent repeatedly, and sumfps-1=71;

[0179] …

[0180] 4. Source frame rate srcfps<1, destination frame rate dstfps<1,

[0181] (1) srcfps>dstfps, encode and output according to the destination frame rate dstfps, such as srcfps=1 / M, dstfps=1 / N, filter each frame in the code stream YUV buffer of the source frame rate srcfps to determine whether to send it to subsequent processing. The filtering rule for each frame is: sumfps+=M, judge sumfps>=N, if it is TRUE, the current frame is selected for subsequent processing, and sumfps-=N, if it is FALSE, the current frame is discarded.

[0182] For example: the source frame rate srcfps is 1 / 4, the destination frame rate dstfps is 1 / 16,

[0183] At the first frame, sumfps+4=4, sumfps<16, and the current frame is discarded (frame loss processing);

[0184] At the second frame, sumfps+4=8, sumfps<16, and the current frame is discarded;

[0185] At the third frame, sumfps+4=12, sumfps<16, and the current frame is discarded;

[0186] At the 4th frame, sumfps+4=16, sumfps>=16, the current frame is retained (sent to frame processing separately), and sumfps-16=0;

[0187] At the 5th frame, sumfps+4=4, sumfps<16, and the current frame is discarded;

[0188] (2) srcfps <= dstfps, optionally, the code can be output according to the source frame rate srcfps, that is, the source code stream frame rate is not changed. Alternatively, the code can be output according to the destination frame rate dstfps, such as srcfps = 1 / M, dstfps = 1 / N. It is necessary to determine whether each frame in the code stream YUV buffer at the source frame rate srcfps is repeatedly sent for encoding. The repetition rule for each frame is: sumfps + = N, judge sumfps > = M, if it is FALSE, the current frame is repeatedly sent, if it is TRUE, the current frame is sent, and sumfps - = M, then continue to process the next frame.

[0189] For example: the source frame rate srcfps is 1 / 16, the destination frame rate dstfps is 1 / 4,

[0190] For the first frame, sumfps+4=4, sumfps<16, the current frame is sent repeatedly (repeated frame processing);

[0191] The first frame is sent repeatedly sumfps+4=8, sumfps<16, the current frame is sent repeatedly;

[0192] The first frame is sent repeatedly sumfps+4=12, sumfps<16, the current frame is sent repeatedly;

[0193] The first frame is repeatedly sent sumfps+4=16, sumfps>=16, the current frame is sent in (sent frame processing separately), and sumfps-16=0;

[0194] At the second frame, sumfps+4=4, sumfps<16, the current frame is sent repeatedly;

[0195] 3. Coding round-robin scheduling;

[0196] In the related art, the encoding process inputs YUV data, converts the serial processing according to the encoding protocol, and outputs the encoded bitstream. The bitstream can only be processed independently on one CPU, and the encoding speed is limited. The present application can enable multi-threaded (multiple encoding threads) parallel encoding processing, and through the encoding information sharing scheduling (i.e., reusing the encoding mode information of the reference bitstream), the bitstream of a channel can be distributed to multiple CPUs for parallel processing. The balanced control system improves CPU utilization and increases the bitstream encoding processing speed. At the same time, when encoding the input YUV data, this scheme inputs the frame data of the reference bitstream (the reference bitstream encoding format is different from the target format of the software encoding bitstream). The frame data of the reference bitstream (i.e., the image frame in the reference bitstream of the target bitstream that corresponds to the frame) carries the encoding mode information required in the software encoding process. By directly extracting information from the reference bitstream during the software encoding process, the steps of software encoding can be reduced, the complexity of software encoding can be reduced, and the speed of software encoding can be increased. Figure 5 As shown, the specific process of encoding round-robin scheduling is as follows:

[0197] When software encodes a bitstream, the encoding input buffer may be composed of multiple rotating buffers, such as a bitstream frame structure buffer and a reference bitstream buffer, each buffer having N data and corresponding to each other. The bitstream frame structure buffers 1 to N respectively store the frame structure data of the 1st to Nth frames. The reference bitstream buffers 1 to N respectively store the 1st to Nth frames of the reference bitstream output by hardware encoding of the 1st to Nth frames of YUV image data; wherein the reference bitstream is the H264 / H265 bitstream after hardware encoding of the bitstream YUV data of the current frame (this format is only an example and is not limited), carrying the encoding mode information required for bitstream encoding, that is, the reference bitstream buffers 1 to N store the frame image data of the H264 / H265 bitstream after hardware encoding of the frame structure data of the current frame. The bitstream frame structure data + the encoded H264 / H265 bitstream, the encoded output target bitstream, are stored in the encoding output buffers 1 to N respectively.

[0198] In the encoding process of the target bitstream, the encoding mode information may be determined first, and then the target bitstream may be encoded according to the determined encoding mode information to obtain the transcoded image data of the target bitstream. The encoding mode information may be rate-distortion optimization (RDO), which is a technology widely used in video compression algorithms for making the best decision in the encoding process to balance the relationship between bit rate and image quality.

[0199] The encoding process of the current frame may include the following steps:

[0200] 1. Prediction; 2. Transformation; 3. Quantization; 4. Entropy coding.

[0201] During the prediction process, intra-frame prediction processing can be performed according to the optimal block size and optimal prediction mode in the determined coding mode information; and inter-frame prediction processing can be performed according to the optimal inter-frame prediction motion vector and prediction residual in the coding mode information.

[0202] During the transformation process, the transformation process can be performed according to the optimal division method in the coding mode information.

[0203] During the quantization process, quantization processing can be performed according to the optimal quantization value in the coding mode information.

[0204] Afterwards, entropy coding (lossless coding according to the information entropy principle) may be performed when encoding the frame structure data of the current frame, thereby obtaining transcoded image data of the current frame.

[0205] Under the encoding round-robin scheduling mechanism, N frames can be encoded in parallel at the same time to improve the processing efficiency of the bitstream data, and different frames can be allocated to different CPUs for encoding. For example, the CPU resources can be called to the maximum extent in a load balancing mode. If the transcoding device has two CPUs, the first frame is stored in the bitstream frame structure buffer 1 + reference bitstream buffer 1, and the encoding output bitstream is sent to the encoding output buffer 1 (the encoding output buffer is unified as a buffer pool. If the encoding output buffer 1 is occupied, the subsequent encoding output buffers 2..N are detected to see if there are free buffers, and the free encoding output buffers are output), and the encoding process runs on CPU1; the second frame is stored in the bitstream frame structure buffer 2 + reference bitstream buffer 2, and the encoding output bitstream is sent to the encoding output buffer 2, and the encoding process runs on CPU2; the third frame is stored in the bitstream frame structure buffer 3 + reference bitstream buffer 3, and the encoding output bitstream is sent to the encoding output buffer 3, and the encoding process runs on CPU1; and so on.

[0206] When the processing speeds of multiple CPUs are different, multiple code streams can be assigned to different CPUs for processing according to specific application scenarios. For example, the processing performance of each of the 1st to 4th CPUs is 3Mbps, and the processing performance of each of the 5th to 8th CPUs is 1Mbps. When processing multiple code streams, in order to maximize the use of CPU performance, for example, a total of 8 code streams, one solution is: first calculate the processing performance M1~M8 required for each code stream, traverse and calculate the sum of the single-channel processing performance value or the combined multi-channel processing performance value, and then compare it with 4Mbps to obtain the optimal solution for multiple code stream allocation.

[0207] (1) If the value of the required processing performance M3 of the third channel is less than or equal to 4Mbps and the value is closest to 4Mbps, the third channel code stream is allocated to the 5th to 8th CPUs for encoding processing in a load balancing mode, and the other 7 channels of code streams are allocated to the 1st to 4th CPUs for encoding processing in a load balancing mode;

[0208] (2) For example, if the sum of the processing performance required for the 5th and 7th channels, M5+M7, is less than or equal to 4Mbps and the value is closest to 4Mbps, then the 5th and 7th channels are allocated to the 5th to 8th CPUs for encoding processing in a load-balanced mode, and the other 6 channels are allocated to the 1st to 4th CPUs for encoding processing in a load-balanced mode.

[0209] The image data processing method provided by the present application is a technical solution for transcoding round-robin scheduling. The decoding, frame rate control, and encoding mechanism implemented by round-robin scheduling can be used in software-implemented decoding, frame rate control, and encoding (encoding can be software or hardware). Through the round-robin scheduling buffer, balanced control improves CPU utilization, reuses the encoding mode information of the reference bitstream, and improves the encoding processing speed of the bitstream data. In addition, the screening algorithm can be used to control whether the frame structure data before encoding is sent to the encoder for encoding, thereby performing frame rate control of multiple source frame rates by lowering and raising the frame rate.

[0210] The present application also provides a transcoding device, such as Figure 6 As shown, including:

[0211] Memory 601, used for storing computer programs;

[0212] The processor 602 is used to implement any of the above-mentioned image data processing methods when executing the program stored in the memory 601.

[0213] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 602, the communication interface, and the memory 601 communicate with each other via the communication bus.

[0214] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0215] The communication interface is used for communication between the above electronic device and other devices.

[0216] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0217] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0218] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image data processing methods are implemented.

[0219] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the image data processing methods in the above embodiments.

[0220] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0221] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0222] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0223] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for processing image data, characterized in that: Applied to a transcoding device, the method comprises: Writing the image data of each frame in the target code stream into a first buffer in an idle state in the first buffer pool in sequence according to the frame order; wherein different first buffers store image data of different frames, and each first buffer is used to store image data of one frame; For each first buffer storing image data, the image data of the current frame in the first buffer is decoded; the decoded data of the current frame obtained by decoding is written into a second buffer in an idle state in a second buffer pool, the first buffer is set to an idle state, and it is detected whether there is a second buffer that meets the space release condition. In response to detecting the existence, the second buffer that meets the space release condition is set to an idle state; wherein different second buffers store decoded data of different frames, and each second buffer is used to store decoded data of one frame; the space release condition includes storing decoded data of a target frame located before the current frame and meeting the data release condition for the encoding stage of the transcoding process, and the target frame is a frame that is not referenced when decoding the frame after the current frame; For each second buffer storing decoded data, encoding preprocessing is performed on the decoded data stored in the second buffer, so that the decoded data stored in the second buffer is stored in a third buffer in a storage form of a frame structure and in a specified data format, thereby obtaining a third buffer storing frame structure data of one frame of the target code stream; wherein the specified data format is a data format required by the encoding stage of the transcoding process, different third buffers store frame structure data of different frames, and each third buffer is used to store frame structure data of one frame; The frame structure data stored in each third buffer is encoded to obtain transcoded image data of the target code stream.

2. The method according to claim 1, characterized in that The step of performing encoding preprocessing on the decoded data stored in the second buffer so as to store the decoded data stored in the second buffer in a third buffer in a storage form of a frame structure and in a specified data format to obtain a third buffer storing frame structure data of one frame of the target code stream comprises: If the data format of the decoded data in the second buffer is different from the specified data format, the decoded data in the second buffer is converted into a format according to the specified data format, and the converted data is stored in a third buffer in an idle state in the third buffer pool in a frame structure form, so as to obtain a third buffer storing the frame structure data of one frame of the target code stream; wherein after the frame structure data of each third buffer in the third buffer pool is encoded, the third buffer is set to an idle state; The method further comprises: In response to the converted data being stored in a third buffer in an idle state in the third buffer pool in a frame structure, the second buffer is marked as satisfying a data release condition for an encoding phase of a transcoding process.

3. The method according to claim 1, characterized in that The step of performing encoding preprocessing on the decoded data stored in the second buffer so as to store the decoded data stored in the second buffer in a third buffer in a storage form of a frame structure and in a specified data format to obtain a third buffer storing frame structure data of one frame of the target code stream comprises: If the data format of the decoded data in the second buffer is the same as the specified data format, determine the information about the frame structure of the decoded data in the second buffer, write the determined frame structure information and a pointer for indicating the address of the second buffer into the target memory area, use the target memory area as a third buffer, and obtain a third buffer storing the frame structure data of one frame of the target code stream; wherein the information about the frame structure is the structure information required to be used when the decoded data in the second buffer is encoded; After the frame structure data in the obtained third buffer is encoded, the second buffer is marked as satisfying the data release condition for the encoding stage of the transcoding process.

4. The method according to any one of claims 1 to 3, characterized in that: The encoding process of the frame structure data stored in each third buffer to obtain the transcoded image data of the target code stream includes: The frame structure data stored in each third buffer is distributed to the target CPU according to the frame sequence, so that the target CPU calls the encoder to encode the received frame structure data to obtain the transcoded image data of the target code stream.

5. The method according to claim 4, characterized in that The frame structure data stored in each third buffer is distributed to the target CPU according to the frame sequence, including: Allocating the frame structure data stored in each third buffer to the target CPU according to the frame sequence and the processing method for frame rate control corresponding to the frame structure data of each frame; The processing mode for frame rate control corresponding to the frame structure data of each frame is determined based on the value of the source frame rate, the value of the destination frame rate and the size relationship between the source frame rate and the destination frame rate, and the processing mode includes frame dropping processing, single frame sending processing or repeated frame sending processing; wherein, if the frame structure data of any frame is subjected to repeated frame sending processing, before the repeated frame sending processing is completed, the second buffer for storing the decoded data of the frame is not set to an idle state; The source frame rate is the frame rate of the target code stream before transcoding, and the destination frame rate is the frame rate required by the code stream after transcoding.

6. The method according to claim 5, characterized in that The process of determining the processing method for frame rate control corresponding to the frame structure data of each frame includes: For the frame structure data of any frame, if the values ​​of the source frame rate and the destination frame rate are not less than the target value, the first control strategy is adopted to determine the processing method of the frame; If the value of the source frame rate is not less than the target value, and the value of the destination frame rate is less than the target value, the second control strategy is adopted to determine the processing method of the frame; If the value of the source frame rate is less than the target value, and the value of the destination frame rate is not less than the target value, the third control strategy is adopted to determine the processing method of the frame; If the values ​​of the source frame rate and the destination frame rate are both less than the target value, the fourth control strategy is adopted to determine the processing method of the frame; The first control strategy, the second control strategy, the third control strategy and the fourth control strategy are all control strategies for adapting the encoding frame rate in the encoding stage of the transcoding process to the target frame rate.

7. The method according to claim 6, characterized in that The first control strategy includes: If the value of the source frame rate is greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the value of the destination frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the value of the source frame rate, and perform a separate frame sending process or a frame drop process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the value of the source frame rate, and update the parameter value of the specified parameter according to the calculated difference; the initial value of the specified parameter is 0; If the value of the source frame rate is not greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the value of the source frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the value of the destination frame rate, and perform a separate frame sending process or a repeated frame sending process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the value of the destination frame rate, and update the parameter value of the specified parameter according to the calculated difference; The second control strategy includes: Calculate the sum of the current parameter value and the target value of the specified parameter, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the first value, and perform a separate frame sending process or a frame drop process on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to a separate frame sending process, calculate the difference between the current parameter value of the specified parameter and the first value, and update the parameter value of the specified parameter according to the calculated difference; the first value is the ratio of the value of the source frame rate to the value of the destination frame rate; The third control strategy includes: Calculate the sum of the current parameter value of the specified parameter and the second value, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the target value, and perform repeated frame sending processing on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is repeatedly sent, calculate the difference between the current parameter value of the specified parameter and the target value, and update the parameter value of the specified parameter according to the calculated difference; the second value is the ratio of the value of the target frame rate to the value of the source frame rate; The fourth control strategy includes: If the value of the source frame rate is greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the magnitude relationship between the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and perform frame drop processing or separate frame sending processing on the frame structure data of the frame according to the determination result; and when the frame structure data of the frame is subjected to separate frame sending processing, calculate the difference between the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and update the parameter value of the specified parameter according to the calculated difference; If the value of the source frame rate is not greater than the value of the destination frame rate, calculate the sum of the current parameter value of the specified parameter and the reciprocal of the value of the destination frame rate, and update the parameter value of the specified parameter based on the calculated sum; determine the relationship between the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and perform repeated frame sending or separate frame sending on the frame structure data of the frame according to the determination result; and when performing separate frame sending on the frame structure data of the frame, calculate the difference between the current parameter value of the specified parameter and the reciprocal of the value of the source frame rate, and update the parameter value of the specified parameter according to the calculated difference.

8. The method according to claim 4, characterized in that The target CPU is at least one CPU allocated to the target code stream according to performance from among multiple CPUs.

9. The method according to claim 1, characterized in that: The frame referenced when decoding any frame is an I frame, and / or a video frame before the frame that has a motion reference with the frame.

10. A transcoding device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-9 when executing a program stored in a memory.

Images