Image data processing method and image data processing device

Abstract

The embodiment of the present application provides an image data processing method and an image data processing device, which relates to the field of data processing technology. The method is applied to an image data processing device, including: obtaining decoded data of each frame obtained by decoding a target code stream; performing auxiliary coding processing; wherein the auxiliary coding processing is: by calling a target encoder module, the decoded data of each frame is encoded according to a reference code stream format to obtain a reference code stream; the coding mode information used when any frame is encoded into a reference code stream format is the same as the coding mode information used when the frame is encoded into a target code stream format; determining the first coding mode information corresponding to each frame to obtain the reference coding mode information of each frame; encoding the decoded data of each frame according to the obtained reference coding mode information to obtain the image data of each frame after encoding in the target code stream format. This solution can improve the efficiency of the code stream encoding process.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an image data processing method and an image data processing device. Background Art

[0002] In the field of image processing, before the bitstream is connected to the target device such as storage device, display device for processing, there is usually a need to transcode the bitstream to meet the target device's access requirements for resolution, frame rate, bit rate, etc. The so-called transcoding is to decode the bitstream first, and then encode the decoded data of each frame obtained by decoding to obtain a bitstream with a custom format or parameters (resolution, frame rate and bit rate), that is, to obtain a bitstream with the target bitstream format.

[0003] With the rapid development of science and technology, the requirements for coding efficiency are getting higher and higher. The current encoding process of code stream is slow and time-consuming; therefore, there is an urgent need for an image data processing method to improve the efficiency of the code stream encoding process. Summary of the invention

[0004] The purpose of the embodiments of the present application is to provide an image data processing method and an image data processing device to improve the efficiency of the encoding process of the code stream. The specific technical solution is as follows:

[0005] In a first aspect, the present application provides an image data processing method, which is applied to an image data processing device, and the method comprises:

[0006] Obtaining decoded data of each frame obtained by decoding the target code stream;

[0007] Perform auxiliary encoding processing; wherein the auxiliary encoding processing is: by calling the target encoder module, the decoded data of each frame is encoded according to the reference bitstream format to obtain the reference bitstream processing; the encoding mode information used when any frame is encoded into the reference bitstream format is the same as the encoding mode information used when the frame is encoded into the target bitstream format;

[0008] Determine the first coding mode information corresponding to each frame, and obtain reference coding mode information of each frame; wherein the first coding mode information corresponding to any frame is the coding mode information of the image frame corresponding to the frame in the reference bitstream;

[0009] According to the acquired reference coding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target code stream format after encoding.

[0010] Optionally, before performing the auxiliary encoding process, the method further includes:

[0011] Performing encoding preprocessing on the decoded data of each frame to obtain frame structure data of each frame; wherein the frame structure data of any frame is data in a specified data format, and the storage form is the storage form of the frame structure; the specified data format is the data format required for encoding;

[0012] The auxiliary encoding process specifically includes: encoding the frame structure data of each frame according to the reference bitstream format to obtain the reference bitstream by calling the target encoder module;

[0013] The step of encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame in the target bitstream format after encoding includes:

[0014] According to the acquired reference coding mode information, the frame structure data of each frame is encoded to obtain the image data of each frame in the target code stream format after encoding.

[0015] Optionally, the method further comprises:

[0016] In response to receiving an instruction to increase the image data processing speed, parameter information of a specified parameter used by the target encoder module during encoding is adjusted; wherein the specified parameter is a parameter related to encoding complexity.

[0017] Optionally, a method for determining the first coding mode information corresponding to any frame includes:

[0018] Acquire, from the coding mode information about each image frame of the reference bitstream, the coding mode information of the image frame corresponding to the frame, and obtain the first coding mode information corresponding to the frame;

[0019] The method for determining the coding mode information of each image frame of the reference bitstream includes:

[0020] Decoding the image data of each image frame in the reference bitstream to obtain the coding mode information about each image frame of the reference bitstream; or receiving the coding mode information of each image frame determined in the coding process of the reference bitstream transmitted by the target encoder module.

[0021] Optionally, the method further comprises:

[0022] Identify whether the target code stream format and the original code stream format meet a matching condition; wherein the matching condition includes: the encoding mode information used when any frame is encoded into the original code stream format is the same as the encoding mode information used when the frame is encoded into the target code stream format; the original code stream format is the code stream format of the target code stream;

[0023] If the matching condition is not met, executing the step of determining the first coding mode information corresponding to each frame to obtain reference coding mode information of each frame;

[0024] If the matching condition is met, the second coding mode information corresponding to each frame is determined, the reference coding mode information of each frame is obtained, and the step of encoding the decoded data of each frame according to the obtained reference coding mode information is performed to obtain the image data of each frame after encoding in the target code stream format; wherein the second coding mode information corresponding to any frame is obtained after decoding the image data of the frame.

[0025] Optionally, encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the encoded image data of each frame in a target bitstream format includes:

[0026] According to the prediction information in the obtained reference coding mode information, intra-frame prediction processing and inter-frame prediction processing are performed on the decoded data of each frame;

[0027] According to the transformation information in the obtained reference coding mode information, transforming the data after the intra-frame prediction processing and the inter-frame prediction processing;

[0028] quantizing the transformed data according to the quantization information in the obtained reference coding mode information;

[0029] According to the entropy coding information in the obtained reference coding mode information, entropy coding is performed on the quantized data.

[0030] Optionally, the performing encoding preprocessing on the decoded data of each frame to obtain frame structure data of each frame includes:

[0031] If the decoded data of each frame is different from the specified data format, for any frame, the decoded data of the frame is converted into a format according to the specified data format, and the converted data is stored in a frame structure buffer in an idle state in the frame structure buffer pool in the form of a frame structure; wherein different frame structure buffers store frame structure data of different frames, and each frame structure buffer is used to store frame structure data of one frame;

[0032] If the decoded data of each frame is in the same format as the specified data, for any frame, information about the frame structure of the decoded data of the frame is determined, the determined frame structure information and a pointer for indicating the address of the decoded buffer of the frame are written into the target memory area, and the target memory area is used as a frame structure buffer; wherein the information about the frame structure of the decoded data of the frame is: structure information required to be used when the decoded data of the frame is encoded; the decoded buffer of the frame is used to store the decoded data of the frame;

[0033] The frame structure data stored in each frame structure buffer is determined as the frame structure data of each frame.

[0034] Optionally, the decoding of the image data of each image frame in the reference bitstream to obtain the coding mode information about each image frame of the reference bitstream includes:

[0035] Decoding the image data of the image frames in each reference code stream buffer to obtain coding mode information about each image frame of the reference code stream; wherein different reference code stream buffers store image data of different image frames in the reference code stream, and each reference code stream buffer is used to store image data of one image frame in the reference code stream.

[0036] Optionally, encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the encoded image data of each frame in a target bitstream format includes:

[0037] The decoded data of each frame is distributed to the target CPU in frame order, so that the target CPU calls the specified encoder module, encodes the decoded data of each frame according to the obtained reference encoding mode information, and obtains the image data of each frame after encoding in the target code stream format.

[0038] In a second aspect, the present application provides an image data processing device, comprising:

[0039] Memory, used to store computer programs;

[0040] The processor is used to implement any of the above-mentioned image data processing methods when executing the program stored in the memory.

[0041] An embodiment of the present application also provides a computer program product comprising instructions, which, when executed on a computer, enables the computer to execute any of the above-mentioned image data processing methods.

[0042] Beneficial effects of the embodiments of the present application:

[0043] The image data processing method provided in the embodiment of the present application is applied to an image data processing device, firstly obtaining decoded data of each frame obtained by decoding a target code stream, and then performing auxiliary encoding processing, that is, calling a target encoder module, encoding the decoded data of each frame according to a reference code stream format to obtain a reference code stream; the encoding mode information used when any frame is encoded into a reference code stream format is the same as the encoding mode information used when the frame is encoded into a target code stream format. Since the encoding mode information is the same, the encoding mode information of the reference code stream can be directly reused for encoding later; specifically, the encoding mode information of the image frame corresponding to each frame in the reference code stream can be determined, that is, the first encoding mode information corresponding to each frame can be determined to obtain the reference encoding mode information of each frame; then, according to the obtained reference encoding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target code stream format. Compared with the method in the prior art that requires a lot of time to calculate the encoding mode information of each frame of the target code stream, the encoding mode information used when any frame is encoded as a reference code stream format in the present application is the same as the encoding mode information used when the frame is encoded as a target code stream format. Therefore, the encoding mode information of each image frame in the reference code stream can be reused to encode the decoded data of each frame. It can be seen that the present application does not need to calculate the encoding mode information of each frame of the target code stream, and can determine the first encoding mode information corresponding to each frame, obtain the reference encoding mode information of each frame, and reuse the reference encoding mode information of each frame to encode the decoded data of each frame; thus, the efficiency of the encoding process of the code stream can be improved through the present application.

[0044] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0046] Figure 1 A schematic diagram of a flow chart of an image data processing method provided in an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a decoding round-robin scheduling process in an image data processing method provided in an embodiment of the present application;

[0048] Figure 3 A schematic diagram of a flow chart of encoding rotation scheduling in an image data processing method provided in an embodiment of the present application;

[0049] Figure 4 A block diagram of an image data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0050] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0051] In the related art, when encoding a bitstream, it is necessary to first calculate the encoding mode information of each frame (such as the encoding mode information determined by the rate-distortion optimization decision) for the decoded data of each frame obtained by the bitstream decoding process, and the calculation of the encoding mode information consumes a lot of time; therefore, the encoding process of the bitstream in the related art is slow and time-consuming, that is, the efficiency of the bitstream encoding process is low.

[0052] Based on this, the embodiments of the present application provide an image data processing method and an image data processing device to improve the efficiency of the encoding process of the code stream.

[0053] The following first introduces an image data processing method provided by the present application.

[0054] Among them, an image data processing method provided by the present application can be applied to an image data processing device, and the image data processing device can be an electronic device with at least an encoding function, such as a gateway device; and the image data processing device can be an encoding device with an encoding function, or a transcoding device with decoding and encoding functions. The present application does not limit the specific form of the image data processing device. In addition, an image data processing method provided by an embodiment of the present application can be applied to any scenario with a code stream encoding requirement, for example: in a scenario where the decoded data of each frame obtained by the code stream decoding process is encoded, after encoding, a code stream (image data) with a target code stream format can be obtained.

[0055] Among them, an image data processing method provided in an embodiment of the present application is applied to an image data processing device, and the method includes:

[0056] Obtaining decoded data of each frame obtained by decoding the target code stream;

[0057] Perform auxiliary encoding processing; wherein the auxiliary encoding processing is: by calling the target encoder module, the decoded data of each frame is encoded according to the reference bitstream format to obtain the reference bitstream processing; the encoding mode information used when any frame is encoded into the reference bitstream format is the same as the encoding mode information used when the frame is encoded into the target bitstream format;

[0058] Determine the first coding mode information corresponding to each frame, and obtain reference coding mode information of each frame; wherein the first coding mode information corresponding to any frame is the coding mode information of the image frame corresponding to the frame in the reference bitstream;

[0059] According to the acquired reference coding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target code stream format after encoding.

[0060] The image data processing method provided in the embodiment of the present application is applied to an image data processing device, firstly obtaining decoded data of each frame obtained by decoding a target code stream, and then performing auxiliary encoding processing, that is, calling a target encoder module, encoding the decoded data of each frame according to a reference code stream format to obtain a reference code stream; the encoding mode information used when any frame is encoded into a reference code stream format is the same as the encoding mode information used when the frame is encoded into a target code stream format. Since the encoding mode information is the same, the encoding mode information of the reference code stream can be directly reused for encoding later; specifically, the encoding mode information of the image frame corresponding to each frame in the reference code stream can be determined, that is, the first encoding mode information corresponding to each frame can be determined to obtain the reference encoding mode information of each frame; then, according to the obtained reference encoding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target code stream format. Compared with the method in the prior art that requires a lot of time to calculate the encoding mode information of each frame of the target code stream, the encoding mode information used when any frame is encoded as a reference code stream format in the present application is the same as the encoding mode information used when the frame is encoded as a target code stream format. Therefore, the encoding mode information of each image frame in the reference code stream can be reused to encode the decoded data of each frame. It can be seen that the present application does not need to calculate the encoding mode information of each frame of the target code stream, and can determine the first encoding mode information corresponding to each frame, obtain the reference encoding mode information of each frame, and reuse the reference encoding mode information of each frame to encode the decoded data of each frame; thus, the efficiency of the encoding process of the code stream can be improved through the present application.

[0061] An image data processing method provided in an embodiment of the present application is exemplarily introduced below in conjunction with the accompanying drawings.

[0062] like Figure 1As shown, an image data processing method provided in an embodiment of the present application, applied to an image data processing device, may include the following steps:

[0063] S101: Obtaining decoded data of each frame obtained by decoding a target bitstream;

[0064] In order to realize the encoding of the target code stream, first, the decoded data of each frame obtained by decoding the target code stream can be obtained; wherein, the process of decoding the target code stream can be similar to the prior art, for example: the image data of each frame of the target code stream is allocated to the CPU according to the frame sequence, so that the CPU calls the decoder to decode the image data of each frame, etc., which will not be elaborated here.

[0065] It is understandable that the decoding process of the image data of any frame may need to rely on the decoded data of other frames as a reference (the image data of each frame can be decoded in frame order), and the frame referenced when decoding any frame is an I frame, and / or a video frame before the frame that has a motion reference with the frame. It is understandable that the frame referenced when decoding any P frame (forward predictive-coded frame, predictive-frame) is the I frame (intra-coded frame, Iframe) before the frame. Of course, any P frame may also refer to a video frame that has a motion reference with the P frame before the P frame to facilitate the decoding of the P frame. In addition, for different bitstream formats, the reference frames may be different. For example, in the H265 bitstream (the decoded data of the bitstream after decoding is in YUV (Y represents brightness, U and V represent chrominance and concentration respectively) format), the frame referenced when the current frame is decoded is the I frame, or the previous video frame of the frame that has a motion reference with the frame (for example, the current frame is the fourth frame, and the third frame may be referenced); in the smart (a video encoding and decoding technology that can be applied to network cameras, video recorders, etc.) format or SVAC (Scalable Video Coding, a video encoding standard) format (the decoded data before decoding is in smart format or SVAC format), the frame referenced when the current frame is decoded is the I frame, or the video frame that has a motion reference with the frame and has a fixed interval with the frame (for example, the current frame is the fourth frame, and the second frame may be referenced).

[0066] S102: Execute auxiliary encoding processing;

[0067] The auxiliary encoding process is: by calling the target encoder module, the decoded data of each frame is encoded according to the reference bitstream format to obtain the reference bitstream process; the encoding mode information used when any frame is encoded into the reference bitstream format is the same as the encoding mode information used when the frame is encoded into the target bitstream format;

[0068] In the present application, it is considered that the encoding mode information used when the same image data is encoded into different bitstream formats may be the same. For example, the encoding mode information used when any frame is encoded into a reference bitstream format (such as the H265 format) is the same as the encoding mode information used when the frame is encoded into a target bitstream format (such as the SVAC format); therefore, auxiliary encoding processing can be performed first, that is, by calling the target encoder module to encode the decoded data of each frame according to the reference bitstream format to obtain a reference bitstream. It should be noted that the target encoder module can be a hardware encoder (of course it can also be a software encoder, and this application does not limit this). The time consumed by the hardware encoder to encode the decoded data of each frame (for example: by calling the GPU to quickly encode the decoded data of each frame) to obtain a reference bit stream (which can be understood as implicitly containing encoding mode information) is greatly reduced compared to the time consumed by the related technology to calculate the encoding mode information of each frame; therefore, this application first performs auxiliary encoding processing to obtain a reference bit stream, and then obtains the corresponding encoding mode information from the reference bit stream for subsequent encoding, thereby reducing the time consumed in the entire encoding process, thereby improving the efficiency of the bit stream encoding process.

[0069] Optionally, in order to facilitate auxiliary encoding processing and subsequent encoding processing, before performing the auxiliary encoding processing, the method further includes:

[0070] Performing encoding preprocessing on the decoded data of each frame to obtain frame structure data of each frame; wherein the frame structure data of any frame is data in a specified data format, and the storage form is the storage form of the frame structure; the specified data format is the data format required for encoding;

[0071] The auxiliary encoding process specifically includes: encoding the frame structure data of each frame according to the reference bitstream format to obtain the reference bitstream by calling the target encoder module;

[0072] The step of encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame in the target bitstream format after encoding includes:

[0073] According to the acquired reference coding mode information, the frame structure data of each frame is encoded to obtain the image data of the target code stream format after encoding each frame; wherein the speed of encoding to generate the reference code stream is positively correlated with the speed of encoding to generate the image data of the target code stream format.

[0074] Before performing auxiliary encoding processing, the decoded data of each frame can be pre-encoded to obtain the frame structure data of each frame; the frame structure data of any frame is data in a specified data format required for encoding, and the storage form is the storage form of the frame structure, and the storage form of the frame structure is also to adapt to the encoding process; thus, during the auxiliary encoding processing, the target encoder module can be called to encode the frame structure data of each frame in accordance with the reference code stream format to obtain a reference code stream; and subsequently, the frame structure data of each frame can be encoded according to the obtained reference encoding mode information to obtain the image data of each frame after encoding in the target code stream format.

[0075] Optionally, the method further comprises:

[0076] In response to receiving an instruction to increase the image data processing speed, parameter information of a specified parameter used by the target encoder module during encoding is adjusted; wherein the specified parameter is a parameter related to encoding complexity.

[0077] It should be noted that, since the coding mode information of the reference code stream directly affects the coding process of the image data in the target code stream format, the faster the speed of coding to generate the reference code stream, the faster the speed of coding to generate the image data in the target code stream format. The present application can also improve the image data processing speed (i.e., improve the coding speed), for example: a user or other device can issue an instruction to improve the image data processing speed, and the image data processing device can respond to the instruction and adjust the parameter information of the specified parameter used when the target encoder module encodes; the specified parameter is a parameter related to the coding complexity, such as: resolution parameter, image size parameter, etc. (or the complexity or requirement of prediction, transformation, quantization and / or entropy coding, etc.); the parameter information of the resolution parameter or image size parameter can be reduced to reduce the complexity of coding to generate the reference code stream, thereby improving the speed of coding to generate the reference code stream, and then, improving the speed of coding to generate the image data in the target code stream format (i.e., improving the image data processing speed).

[0078] Exemplarily, in one implementation, the encoding preprocessing is performed on the decoded data of each frame to obtain the frame structure data of each frame, including:

[0079] If the decoded data of each frame is different from the specified data format, for any frame, the decoded data of the frame is converted into a format according to the specified data format, and the converted data is stored in a frame structure buffer in an idle state in the frame structure buffer pool in the form of a frame structure; wherein different frame structure buffers store frame structure data of different frames, and each frame structure buffer is used to store frame structure data of one frame;

[0080] If the decoded data of each frame is in the same format as the specified data, for any frame, information about the frame structure of the decoded data of the frame is determined, the determined frame structure information and a pointer for indicating the address of the decoded buffer of the frame are written into the target memory area, and the target memory area is used as a frame structure buffer; wherein the information about the frame structure of the decoded data of the frame is: structure information required to be used when the decoded data of the frame is encoded; the decoded buffer of the frame is used to store the decoded data of the frame;

[0081] The frame structure data stored in each frame structure buffer is determined as the frame structure data of each frame.

[0082] In the present application, the decoded data of each frame is the same as or different from the specified data format, and the decoded data of each frame is encoded and preprocessed in different ways to obtain the frame structure data of each frame.

[0083] Specifically, if the decoded data of each frame is different from the specified data format, format conversion is required first; for any frame, the decoded data of the frame is format converted according to the specified data format (that is, the decoded data format of the frame is converted into data in the specified data format), and the converted data is stored in the form of a frame structure in an idle frame structure buffer in the frame structure buffer pool; different frame structure buffers store frame structure data of different frames, and each frame structure buffer is used to store the frame structure data of one frame. After the frame structure data stored in any frame structure buffer is encoded into image data in the target code stream format, the frame structure buffer is released to an idle state to facilitate the storage of frame structure data of other frames, thereby improving the efficiency of subsequent encoding processes.

[0084] If the decoded data of each frame is the same as the specified data format, for any frame, the frame structure information of the decoded data of the frame is determined, and the determined frame structure information and the pointer used to indicate the address of the decoding buffer of the frame are written into the target memory area, and the target memory area is used as a frame structure buffer; thereafter, the frame structure data stored in each frame structure buffer can be determined as the frame structure data of each frame.

[0085] Among them, the decoding buffer of any frame is used to store the decoding data of the frame, different decoding buffers are used to store the decoding data of different frames, and each decoding buffer is used to store the decoding data of one frame; if the decoding data of each frame is different from the specified data format, for any frame, it is necessary to first perform format conversion according to the specified data format, and store the converted data in the form of a frame structure in an idle frame structure buffer (the decoding buffer of the frame is not reused at this time); if the decoding data of each frame is the same as the specified data format, for any frame, the decoding buffer of the frame can be mapped to a frame structure buffer (that is, in the encoding process, multiplexing or That is to say, occupying the decoding buffer), the information about the frame structure of the decoding data of the frame can be determined, and the determined frame structure information and the pointer used to indicate the address of the decoding buffer of the frame (equivalent to the target memory area containing the decoding buffer of the frame and the decoding data stored in the decoding buffer) are written into the target memory area, and the target memory area is used as a frame structure buffer; at this time, the decoding buffer of the frame is occupied by the encoding process, and when the frame structure data of the decoding buffer of the frame in the obtained frame structure buffer is encoded, the decoding buffer of the frame can be released to an idle state to facilitate the storage of decoding data of other frames.

[0086] In addition, it should be emphasized that if the decoded data of each frame is the same as the specified data format, for any frame, the decoded data of the frame can also be stored in the form of a frame structure in a frame structure buffer in an idle state in the frame structure buffer pool (no format conversion is required at this time), and the present application does not limit this. Of course, if the decoded data of each frame is the same as the specified data format, by mapping the decoded buffer to a frame structure buffer (that is, writing the information of the determined frame structure and the pointer for indicating the address of the decoded buffer of the frame into the target memory area, and using the target memory area as a frame structure buffer), the occupancy of the frame structure buffer in the frame structure buffer pool can be reduced, and the efficiency of the encoding process can be improved. Moreover, when the decoded data of each frame is different from the specified data format, when the format-converted data is stored in the form of a frame structure in a frame structure buffer in an idle state in the frame structure buffer pool, the information about the frame structure of the format-converted data can also be determined to adapt to the encoding requirements.

[0087] It should be noted that there are multiple decoding buffers in the decoding buffer pool. Due to the limited number of decoding buffers, there may be some frames whose image data has not been decoded and has not been written to a decoding buffer in an idle state (after decoding each frame of the target code stream, the decoding data of each frame is written to a decoding buffer in an idle state). If there is no decoding buffer in an idle state, you can wait until there is a decoding buffer in an idle state in the decoding buffer pool, continue to decode the image data of the remaining frames in the target code stream in frame order, and write the obtained decoding data of one frame to a decoding buffer in an idle state. Of course, the number of decoding buffers may also be greater than the number of decoding data of each frame in the target code stream, so that after writing the decoding data of each frame of the target code stream, there may still be one or more decoding buffers in an idle state in the decoding buffer pool, which is reasonable.

[0088] Exemplarily, the information about the frame structure includes: information about the width and height of the frame, the format of the frame, the width and height span of the frame, frame coding and / or field coding. It should be noted that due to the different specified data formats required for the encoding process, the required information about the frame structure can be flexibly adjusted, and the information about the frame structure can also include other information about the frame structure of the encoding process. Under the condition of meeting the requirements of the encoding process, this application does not limit the information about the frame structure.

[0089] Of course, regardless of whether it is necessary to map the decoding buffer to a frame structure buffer, the method of storing the decoded data of the decoding buffer in the form of a frame structure can be to fill in the information about the frame structure and then store it; if the data format of the decoded data in the decoding buffer is different from the specified data format, that is, the decoding buffer does not need to be mapped to a frame structure buffer, then the decoded data in the decoding buffer needs to be format converted according to the specified data format, and then the converted data is filled with information about the frame structure. This application does not limit this.

[0090] It should be noted that there are multiple frame structure buffers in the frame structure buffer pool, each frame structure buffer is used to store the frame structure data of a frame, and different frame structure buffers store the frame structure data of different frames. Due to the limited number of frame structure buffers, there may be decoded data of some frames that have not been pre-encoded. It is possible to wait for the existence of an idle frame structure buffer in the frame structure buffer pool, and then continue to perform encoding pre-processing on the decoded data of the remaining frames in the target code stream in frame order, and store them in the idle frame structure buffer. Of course, the number of frame structure buffers may also be greater than the number of decoded data of each frame in the target code stream, so that after storing the frame structure data of each frame of the target code stream, there may still be one or more idle frame structure buffers in the frame structure buffer pool, which is reasonable.

[0091] S103: Determine first coding mode information corresponding to each frame, and obtain reference coding mode information of each frame;

[0092] The first coding mode information corresponding to any frame is the coding mode information of the image frame corresponding to the frame in the reference bitstream;

[0093] After the auxiliary coding process is performed, for subsequent coding, the first coding mode information of the image frame corresponding to each frame in the reference code stream may be determined to obtain the reference coding mode information of each frame.

[0094] Exemplarily, in one implementation, a method for determining the first coding mode information corresponding to any frame includes:

[0095] Acquire, from the coding mode information about each image frame of the reference bitstream, the coding mode information of the image frame corresponding to the frame, and obtain the first coding mode information corresponding to the frame;

[0096] The method for determining the coding mode information of each image frame of the reference bitstream includes:

[0097] Decoding the image data of each image frame in the reference bitstream to obtain the coding mode information about each image frame of the reference bitstream; or receiving the coding mode information of each image frame determined in the coding process of the reference bitstream transmitted by the target encoder module.

[0098] When the first coding mode information corresponding to any frame is determined, the coding mode information of the image frame corresponding to the frame can be obtained from the coding mode information about each image frame in the reference code stream, so as to obtain the first coding mode information corresponding to the frame; wherein, the image data of each image frame in the reference code stream can be decoded to obtain the coding mode information about each image frame of the reference code stream; or, the target encoder module can transmit the coding mode information of each image frame determined in the encoding process of the reference code stream to the image data processing device, and can directly receive the coding mode information of each image frame transmitted by the target encoder.

[0099] Among them, although it is necessary to perform auxiliary encoding processing to obtain a reference code stream in the present application, and obtain the encoding mode information of the image frame corresponding to the frame from the encoding mode information about each image frame in the reference code stream, and obtain the first encoding mode information corresponding to the frame (such as decoding the image data of each image frame in the reference code stream to obtain the encoding mode information of each image frame, or receiving the encoding mode information of each image frame transmitted by the target encoder module); although the hardware encoding (i.e., auxiliary encoding processing) and decoding processes are added, compared with the specified encoder module (which can be a software encoder) calculating the encoding mode information of each frame during the encoding process and encoding according to the encoding mode information, a lot of time is saved.

[0100] In addition, the target hardware encoder that encodes the reference bit stream can also transmit the reference coding mode information corresponding to each frame determined during the encoding process for direct multiplexing without decoding to obtain the reference coding mode information corresponding to the frame, thereby further improving the encoding efficiency.

[0101] Specifically, the decoding process of the image data of each image frame in the reference bitstream to obtain the coding mode information about each image frame of the reference bitstream includes:

[0102] Decoding the image data of the image frames in each reference code stream buffer to obtain coding mode information about each image frame of the reference code stream; wherein different reference code stream buffers store image data of different image frames in the reference code stream, and each reference code stream buffer is used to store image data of one image frame in the reference code stream.

[0103] The present application also provides a reference code stream buffer, where different reference code stream buffers store image data of different image frames in the reference code stream, and each reference code stream buffer is used to store image data of an image frame in the reference code stream; the image data of the image frames in the multiple reference code stream buffers can be decoded to obtain the coding mode information about each image frame of the reference code stream. In addition, there can be a corresponding relationship between the frame structure buffer and the reference code stream buffer, where the frame to which the frame structure data stored in the frame structure buffer belongs and the image frame to which the image data stored in the reference code stream buffer belongs are the corresponding frames in the target code stream and the reference code stream.

[0104] It should be noted that there are multiple reference code stream buffers in the present application. Since the number of reference code stream buffers is limited, there may be image data of some image frames that have not been stored in a reference code stream buffer (each reference code stream buffer may not have an idle reference code stream buffer). When there is an idle reference code stream buffer, the image data of the remaining image frames in the reference code stream can be stored in the idle reference code stream buffer in accordance with the frame sequence. Of course, the number of reference code stream buffers may also be greater than the number of image data of each image frame in the reference code stream, so that after storing the image data of each image frame of the reference code stream, there may still be one or more idle reference code stream buffers, which is reasonable.

[0105] In addition, it should be emphasized that the image frame corresponding to any frame in the reference bitstream can be understood as the image frame obtained after hardware encoding the frame structure data of the frame; the image frame obtained by hardware encoding the frame structure data of the frame can be in H264 / H265 format (i.e., the reference bitstream format). If the image data obtained by encoding the frame structure data of the frame through a soft encoder (i.e., a specified encoder module) is in the target bitstream format, the image frame corresponding to the frame in the reference bitstream and the image data in the target bitstream format after encoding the frame both represent the same content, for example: they both contain the same trees, and the position, posture, background, etc. of the represented trees are the same.

[0106] S104: encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the encoded image data of each frame in the target bitstream format;

[0107] After obtaining the reference coding mode information of each frame, the decoded data of each frame (or the frame structure data after coding preprocessing) can be encoded to obtain the encoded image data of each frame in the target code stream format.

[0108] Exemplarily, encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the encoded image data of each frame in the target code stream format includes:

[0109] According to the prediction information in the obtained reference coding mode information, intra-frame prediction processing and inter-frame prediction processing are performed on the decoded data of each frame;

[0110] According to the transformation information in the obtained reference coding mode information, transforming the data after the intra-frame prediction processing and the inter-frame prediction processing;

[0111] quantizing the transformed data according to the quantization information in the obtained reference coding mode information;

[0112] According to the entropy coding information in the obtained reference coding mode information, entropy coding is performed on the quantized data.

[0113] When encoding the decoded data (or frame structure data) of each frame according to the obtained reference coding mode information, the decoded data of each frame may be subjected to intra-frame prediction processing according to the prediction information in the obtained reference coding mode information (such as: the optimal intra-frame prediction block size and the prediction mode corresponding to each block); and the decoded data of each frame may be subjected to inter-frame prediction processing according to the prediction information in the obtained reference coding mode information (such as: the optimal inter-frame prediction motion vector and the prediction residual). The present application does not limit the order of intra-frame prediction processing and inter-frame prediction processing, and the intra-frame prediction processing may be performed first, and then the inter-frame prediction processing may be performed on the data after the intra-frame prediction processing.

[0114] Afterwards, the data after intra-frame prediction and inter-frame prediction can be transformed according to the transformation information (such as the optimal transformation division method) in the reference coding mode information obtained; and the transformed data can be quantized according to the quantization information (such as the optimal quantization value) in the reference coding mode information obtained. Of course, after the quantization, entropy coding can also be performed, and entropy coding can be performed according to the entropy coding information in the reference coding mode information obtained, so as to obtain the image data in the target code stream format after each frame is encoded.

[0115] In the present application, since the encoding speed of the reference code stream is relatively fast and the calculation of the encoding mode information is also relatively fast, the present application can directly reuse the first encoding mode information of the image frame corresponding to each frame in the reference code stream to obtain the reference encoding mode information of each frame, thereby directly encoding the decoded data of each frame, thereby improving the efficiency of the encoding process of the code stream.

[0116] In addition, encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame after encoding in the target code stream format includes:

[0117] The decoded data of each frame is distributed to the target CPU in frame order, so that the target CPU calls the specified encoder module, encodes the decoded data of each frame according to the obtained reference encoding mode information, and obtains the image data of each frame after encoding in the target code stream format.

[0118] The target CPU is at least one CPU allocated to the target code stream according to performance from among multiple CPUs.

[0119] In this application, N frames can be encoded in parallel at the same time to improve the processing efficiency of encoding, and different frames can be allocated to different CPUs for encoding. For example, a load balancing mode can be used to maximize the use of CPU resources. For example, if there are two CPUs with the same performance in the image data processing device, the encoding of the first frame runs on CPU1; the encoding of the second frame runs on CPU2; the encoding of the third frame runs on CPU1, and so on.

[0120] When the performance of multiple CPUs in the image data processing device is different, multiple code streams can be allocated to different CPUs for processing according to specific application scenarios. For example, if the processing performance of each of the 1st to 4th CPUs is 3Mbps, and the processing performance of each of the 5th to 8th CPUs is 1Mbps, then for multiple code streams, in order to maximize the use of CPU performance, for example, a total of 8 code streams, one solution is: first calculate the processing performance of the CPU required for each code stream, traverse and calculate the sum of the single-channel processing performance value or the combined multi-channel processing performance value, and then compare it with 4Mbps (the processing performance of the last 4 CPUs (5th to 8th CPUs) is 1Mbps × 4), and obtain the optimal solution for multiple code stream allocation.

[0121] For example: if the processing performance required for the 3rd code stream is less than or equal to 4Mbps and the value is closest to 4Mbps, the 3rd code stream is allocated to CPUs 5-8 for encoding in a load-balanced mode, and the other 7 code streams are allocated to CPUs 1-4 for encoding in a load-balanced mode. For another example: if the sum of the processing performance required for the 5th and 7th code streams is less than or equal to 4Mbps and the value is closest to 4Mbps, the 5th and 7th code streams are allocated to CPUs 5-8 for encoding in a load-balanced mode, and the other 6 code streams are allocated to CPUs 1-4 for encoding in a load-balanced mode.

[0122] The target CPU can call a specified encoder module (software encoder, each CPU can call one or more software encoders, and this application does not limit this), encode the decoded data of each frame according to the obtained reference encoding mode information, and obtain the image data of each frame after encoding in the target code stream format.

[0123] It should be emphasized that the image data processing method provided by the present application can perform encoding processing on one or more code streams, and any target code stream can be encoded in the above-mentioned manner, and the target code streams can be encoded in parallel according to the target CPU assigned to each target code stream, thereby improving the encoding efficiency of the code stream.

[0124] In the technical solution of the present application, the operations involved in acquiring, storing, using, processing, transmitting, providing and disclosing decoding data, encoding mode information and image data are all carried out with the user's authorization.

[0125] The image data processing method provided in the embodiment of the present application is applied to an image data processing device, firstly obtaining decoded data of each frame obtained by decoding a target code stream, and then performing auxiliary encoding processing, that is, calling a target encoder module, encoding the decoded data of each frame according to a reference code stream format to obtain a reference code stream; the encoding mode information used when any frame is encoded into a reference code stream format is the same as the encoding mode information used when the frame is encoded into a target code stream format. Since the encoding mode information is the same, the encoding mode information of the reference code stream can be directly reused for encoding later; specifically, the encoding mode information of the image frame corresponding to each frame in the reference code stream can be determined, that is, the first encoding mode information corresponding to each frame can be determined to obtain the reference encoding mode information of each frame; then, according to the obtained reference encoding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target code stream format. Compared with the method in the prior art that requires a lot of time to calculate the encoding mode information of each frame of the target code stream, the encoding mode information used when any frame is encoded as a reference code stream format in the present application is the same as the encoding mode information used when the frame is encoded as a target code stream format. Therefore, the encoding mode information of each image frame in the reference code stream can be reused to encode the decoded data of each frame. It can be seen that the present application does not need to calculate the encoding mode information of each frame of the target code stream, and can determine the first encoding mode information corresponding to each frame, obtain the reference encoding mode information of each frame, and reuse the reference encoding mode information of each frame to encode the decoded data of each frame; thus, the efficiency of the encoding process of the code stream can be improved through the present application.

[0126] Optionally, in another embodiment of the present application, the method further includes:

[0127] Identify whether the target code stream format and the original code stream format meet a matching condition; wherein the matching condition includes: the encoding mode information used when any frame is encoded into the original code stream format is the same as the encoding mode information used when the frame is encoded into the target code stream format; the original code stream format is the code stream format of the target code stream;

[0128] If the matching condition is not met, executing the step of determining the first coding mode information corresponding to each frame to obtain reference coding mode information of each frame;

[0129] If the matching condition is met, the second coding mode information corresponding to each frame is determined, the reference coding mode information of each frame is obtained, and the step of encoding the decoded data of each frame according to the obtained reference coding mode information is performed to obtain the image data of each frame after encoding in the target code stream format; wherein the second coding mode information corresponding to any frame is obtained after decoding the image data of the frame.

[0130] In the present application, it is also possible to first identify whether the target code stream format and the original code stream format meet the matching condition, that is, to determine whether the coding mode information used when the same image data is encoded into the target code stream format and the original code stream format is the same; if the matching condition is not met, that is, they are not the same, then the coding mode information cannot be directly obtained from the target code stream, and it is necessary to perform the step of determining the first coding mode information corresponding to each frame and obtaining the reference coding mode information of each frame (of course, the step of auxiliary coding processing must also be performed). If the matching condition is met, that is, they are the same, then the coding mode information can be directly obtained from the target code stream and reused, and it is not necessary to perform auxiliary coding processing to obtain the reference code stream. The image data of each frame of the target code stream can be decoded, the second coding mode information corresponding to each frame can be determined, and the reference coding mode information of each frame can be obtained. The step of encoding the decoded data of each frame according to the obtained reference coding mode information to obtain the image data of each frame in the target code stream format after encoding is performed to encode the decoded data of each frame.

[0131] It can be seen that in the present application, it is also possible to first identify whether the target code stream format and the original code stream format meet the matching conditions, thereby performing auxiliary encoding processing and multiplexing the encoding mode information of the reference code stream, or directly decode the target code stream and multiplex the encoding mode information of the target code stream; by multiplexing the encoding mode information, the decoded data of each frame can be quickly encoded to improve the efficiency of the encoding process of the code stream.

[0132] Optionally, in another embodiment of the present application, the obtaining decoded data of each frame obtained by decoding the target code stream includes:

[0133] Obtaining decoded data of each frame obtained by decoding the target bitstream from each decoding buffer in the decoding buffer pool;

[0134] Any decoding buffer is further used for: storing the second coding mode information obtained after decoding the image data of the frame when the target code stream format and the original code stream format meet the matching condition;

[0135] The obtaining the second coding mode information corresponding to each frame includes:

[0136] The second encoding mode information corresponding to each frame is obtained from each decoding buffer in the decoding buffer pool.

[0137] In addition, any decoding buffer is associated with a coding mode information buffer, and the coding mode information buffer associated with any decoding buffer is used to store the second coding mode information obtained after decoding the image data of the frame when the target code stream format and the original code stream format meet the matching condition;

[0138] The obtaining the second coding mode information corresponding to each frame includes:

[0139] The second coding mode information corresponding to each frame is obtained from the coding mode information buffer associated with each decoding buffer in the decoding buffer pool.

[0140] In the present application, when the target code stream format and the original code stream format meet the matching conditions, the second coding mode information obtained after decoding the image data of the frame can be directly stored in the decoding buffer of the frame; or, the second coding mode information obtained after decoding the image data of the frame can be stored in the coding mode information buffer associated with the decoding buffer of the frame. The present application does not limit this.

[0141] In the present application, multiple decoding buffers, multiple frame structure buffers, reference code stream buffers and other buffers may be provided. By polling and releasing the stored data in each buffer, the image data of each frame of the code stream may be encoded in parallel. Furthermore, each buffer may be a memory in an image data processing device, which is not limited in the present application.

[0142] An image data processing method provided by the present application is described in detail below in conjunction with a specific embodiment.

[0143] The technical solution of the present application is a technical solution for encoding round-robin scheduling, which uses round-robin scheduling to achieve decoding and encoding control, and can be used in software-implemented round-robin scheduling solutions for decoding and encoding (the encoding process can use software or hardware), thereby improving the processing efficiency of the encoding process of the code stream.

[0144] The present application provides an image data processing method, which can realize transcoding functions including: decoding round-robin scheduling; encoding round-robin scheduling;

[0145] 1. Decoding round-robin scheduling;

[0146] In the decoding process of the present application, threads can be added to process the image data of each frame of the same channel code stream in parallel through round-robin scheduling to improve the decoding speed of the code stream (for example, a decoding thread can be called for decoding each frame, and each frame can be decoded in parallel. The decoding thread can be restored after decoding, or, for a continuous code stream, multiple decoding threads can be continuously called to continuously decode each frame of the code stream); during the decoding process, the previous and next frames of the code stream have a reference dependency relationship, so the decoding buffer of the current frame is delayed to provide information for the decoding of subsequent frames (such as motion reference information, etc.); Figure 2 As shown, the decoding round-robin scheduling process is as follows:

[0147] Decoding input buffers 1 to N (i.e., different decoding input buffers store image data of different frames, each decoding input buffer is used to store image data of one frame, and are all in an idle state by default) in the decoding input buffer pool are polled to input the code stream data to be decoded (i.e., image data of each frame of the target code stream, each decoding input buffer is used to store image data of one frame, and different decoding input buffers store image data of different frames), and the decoded data is stored in decoding buffers 1 to N in the decoding buffer pool after decoding; for the image data of the current frame in any decoding input buffer, a decoding buffer in an idle state in the decoding buffer pool is queried before decoding, and the decoded data after decoding the current frame is input into the idle decoding buffer.

[0148] At the same time, it is detected whether there is an unreleased decoding buffer for the decoded data of the frame before the current frame decoding process, that is, it is detected whether there is a decoding buffer of the frame before the current frame that needs to be released. If not, continue to decode and process the image data of the next frame. Since there is still encoding processing in the later stage (the encoding process may occupy the decoding buffer of the current frame), the decoding buffer of the current frame is not released temporarily. If it exists, continue to judge whether the decoded data of the frame in the unreleased decoding buffer is a reference frame of the subsequent frame (that is, the decoding of the subsequent frame needs to obtain information from the frame in the unreleased decoding buffer, such as: motion reference information), or whether the unreleased decoding buffer is occupied during the encoding process; when the frame in the unreleased decoding buffer is no longer a referenced frame and is no longer used in the encoding process, the decoding buffer is released, and then the decoding buffer becomes idle; when the frame in the unreleased decoding buffer is a reference frame when decoding a subsequent frame, or the subsequent encoding process needs to occupy the decoding buffer, the decoding buffer is not released, and the image data of the next frame is continued to be decoded.

[0149] For each decoding buffer storing decoded data, if the format of the decoded data is the same as the specified data format required by the encoding input, the decoding buffer can be used as a frame structure buffer of the frame structure buffer pool of the encoding process. If the format of the decoded data is different from the specified data format required by the encoding input, format conversion is required before encoding (that is, the decoded data is converted according to the specified data format, and stored in the form of a frame structure). The frame structure buffers 1~N used before encoding are also uniformly a buffer pool. When the frame structure data of the current frame is output (that is, the decoded data of the frame is converted according to the specified data format, and the converted data is stored in the form of a frame structure in an idle frame structure buffer in the frame structure buffer pool), it is traversed and queried whether there is an idle frame structure buffer in the frame structure buffer pool. If there is an idle frame structure buffer, the idle frame structure buffer is occupied, the frame structure data after the format conversion is stored in the frame structure buffer, and then sent for encoding; the used frame structure buffer is released after the encoding process is completed; at this time, the encoding process will not affect the release of the decoding buffer, but only affects the release of the frame structure buffer. After the format conversion of the decoded data in the decoding buffer of the current frame is completed, and it is subsequently determined that the current frame is not a frame referenced when decoding the frame after the current frame, the decoding buffer can be released.

[0150] by Figure 2 Take the example of buffer zone occupation and release in each stage as follows:

[0151] 1. Before bitstream processing, all buffers of the decoding input buffer pool, decoding buffer pool, and frame structure buffer pool are in the released state, and the image data of each frame of the target bitstream is input into the decoding input buffer 1-N in sequence according to the frame order; the image data of the first frame is sent to the decoding input buffer 1, the image data of the second frame is sent to the decoding input buffer 2, and so on, and the image data of the Nth frame is sent to the decoding input buffer N.

[0152] 2. The decoded data of any frame after decoding is placed in the decoding buffer for storage. Assuming that the first frame is an I frame, the decoded data of the image data of the first frame after decoding is placed in decoding buffer 1. Since the subsequent code stream decoding needs to refer to the first frame I frame, after the format of the decoded data of the first frame is converted, when the decoding buffer release detection is performed, decoding buffer 1 is not released, and the second frame processing continues.

[0153] 3. If the second frame is a P frame, the decoded data of the second frame's image data is decoded and placed in decoding buffer 2. Since the decoding of the third frame needs to refer to the first frame I frame and the second frame P frame, after the second frame's decoded data is format converted, when the decoding buffer release detection is performed, neither decoding buffer 1 nor decoding buffer 2 is released, and the processing of the third frame continues.

[0154] 4. If the third frame is a P frame, the decoded data of the image data of the third frame is decoded and placed in decoding buffer 3. Since the decoding of the fourth frame needs to refer to the first frame I frame and the third frame P frame, after the format conversion of the decoded data of the third frame, when the decoding buffer release detection is performed, decoding buffer 2 is released (at this time, whether the encoding process occupies the decoding buffer 2 is not considered. It can be understood that the encoding process does not occupy the decoding buffer 2 by default; or, if the encoding process occupies the decoding buffer 2, the decoding buffer 2 is not directly released, and the subsequent release of the decoding buffer is similar). Neither decoding buffer 1 nor decoding buffer 3 is released, and the processing of the fourth frame continues.

[0155] 5. If the 4th frame is a P frame, the decoded data of the 4th frame after image data decoding is placed in decoding buffer 4. Since the decoding of the 5th frame needs to refer to the 1st frame I frame and the 4th frame P frame, after the format conversion of the decoded data of the 4th frame, when the decoding buffer release detection is performed, decoding buffer 3 is released, and decoding buffer 1 and decoding buffer 4 are not released, and the processing of the 5th frame continues.

[0156] 6. Continue to process the image data of each subsequent frame according to the above rules. For the bitstream in smart or SVAC format, the reference frame relationship is more complicated, and more frames need to be released later (for example: reference I frame and P frame at fixed intervals, such as: the 10th frame refers to the I frame of the first frame and the P frames of the 2nd, 4th, 6th and 8th frames).

[0157] 7. After using decoding buffer N, the processing of the next frame starts from decoding buffer 1 in the decoding buffer pool in order. If decoding buffer 1 is still occupied and not released, the subsequent decoding buffers will continue to be searched until a released decoding buffer (that is, an idle decoding buffer) is found, and the decoding processing of the frame will continue.

[0158] 8. For the decoded data in any decoding buffer, if the decoded data output by decoding does not meet the input requirements of encoding (the data format of the decoded data is different from the specified data format), format conversion processing is performed before encoding (the decoded data of the frame is converted according to the specified data format, and the converted data is stored in the form of a frame structure in an idle frame structure buffer in the frame structure buffer pool). At this time, after the format conversion of the decoded data in the decoding buffer, if the decoded data of the current frame in the decoding buffer is not used as a frame for reference when decoding the frame after the current frame, it can be released immediately. Subsequent encoding processing will only limit the release of the frame structure buffer in the frame structure buffer pool.

[0159] 9. If the decoded data output by the decoder meets the input requirements of the encoding (that is, the data format of the decoded data is the same as the specified data format), the decoded buffer can be directly mapped as the frame structure buffer of the encoding input. At this time, the decoded buffer cannot be released immediately, and it is necessary to detect whether the subsequent encoding process occupies the current decoded buffer. If the encoding process is still using the current decoded buffer, it cannot be released, and the processing of the next frame continues. If the encoding process no longer uses the current decoded buffer, and the decoded data of the current frame in the decoded buffer is not used as a reference frame for decoding the frame after the current frame, the current decoded buffer is judged to be releasable and the buffer is released.

[0160] 2. Coding round-robin scheduling;

[0161] In the related art, the encoding process inputs frame structure data, converts the serial processing according to the encoding protocol, and outputs the encoded bit stream. The bit stream can only be processed independently on one CPU, and the encoding speed is limited. This application can enable multi-threaded (multiple encoding threads) parallel encoding processing, and through the shared scheduling of encoding information (i.e., multiplexing reference encoding mode information), the bit stream of a channel can be distributed to multiple CPUs for parallel processing. The balanced control system improves CPU utilization and increases the bit stream encoding processing speed. At the same time, this solution inputs the frame data of the reference code stream (the reference code stream format is different from the target code stream format of the software encoding code stream) while encoding the input frame structure data. The frame data of the reference code stream (i.e., the image frame in the reference code stream corresponding to the frame) carries the encoding mode information required in the software encoding process. By directly extracting information from the reference code stream during the software encoding process, the steps of software encoding can be reduced, the complexity of software encoding can be reduced, and the efficiency of software encoding can be improved (this embodiment is introduced by taking the first encoding mode information corresponding to each frame as the reference encoding mode information as an example; of course, if the target code stream format and the original code stream format meet the matching conditions, the second encoding mode information corresponding to each frame can also be used as the reference encoding mode information for encoding). Figure 3 As shown, the specific process of encoding round-robin scheduling is as follows:

[0162] When software encodes the bitstream, the encoding input buffer can be composed of multiple rotating buffers, such as a frame structure buffer and a reference bitstream buffer, each buffer having N data and corresponding to each other. The frame structure data of the 1st to Nth frames are stored in the frame structure buffers 1 to N respectively. The reference bitstream buffers 1 to N respectively store the 1st to Nth frames of the reference bitstream output by hardware encoding of the frame structure data of the 1st to Nth frames; wherein the reference bitstream is the H264 / H265 bitstream after hardware encoding of the decoded data of each frame (this format is only an example and is not limited), carrying the encoding mode information required for bitstream encoding, that is, the reference bitstream buffers 1 to N store the frame image data of the H264 / H265 bitstream after hardware encoding of the frame structure data of the current frame. The bitstream frame structure data + the encoded H264 / H265 bitstream, the image data of the encoded output target bitstream format, are stored in the encoding output buffers 1 to N respectively.

[0163] In the encoding process of the target bitstream, the coding mode information of the reference bitstream is directly used (of course, the coding mode information can also be recalculated), such as rate-distortion optimization (RDO). Rate-distortion optimization is a technology widely used in video compression algorithms to make the best decision in the encoding process to balance the relationship between bit rate and image quality. The basic steps of RDO decision are as follows:

[0164] 1. Candidate generation: For each macroblock or coding unit, multiple coding mode candidate schemes are generated, which may include different block sizes, prediction modes, etc.

[0165] 2. Rate-distortion evaluation: For each candidate solution, calculate its encoded bit rate and distortion compared to the original image. Use a rate-distortion cost function to comprehensively evaluate the quality of each candidate solution. The function is usually in the form of J=D+λR, where D is the distortion, R is the bit rate, λ is the Lagrange multiplier, and J is the quality of the candidate solution.

[0166] 3. Decision: Select the candidate with the lowest cost for final encoding based on the rate-distortion cost function. The overall bit rate target is controlled by adjusting the value of λ, thus affecting the final video quality and file size.

[0167] During the encoding process of the target bitstream, there is no need to calculate the rate-distortion optimization decision again to select the mode. The rate-distortion optimization decision of the reference coding mode information can be directly reused, which effectively reduces the steps of software encoding, reduces the complexity of software encoding, and improves the efficiency of software encoding.

[0168] The encoding process can be divided into the following steps:

[0169] 1. Prediction; 2. Transformation; 3. Quantization; 4. Entropy coding.

[0170] During the prediction process, the optimal intra-frame prediction block size optimized by RDO and the prediction mode corresponding to each block of the current frame can be obtained by decoding the reference bitstream. When encoding the frame structure data of the current frame, the predicted optimal block size division method and the optimal prediction mode are no longer calculated. The optimal intra-frame prediction block size of the corresponding image frame in the reference bitstream and the prediction mode corresponding to each block are directly used for intra-frame prediction processing.

[0171] In the prediction process, the reference bitstream can also be decoded to obtain the optimal inter-frame prediction motion vector and prediction residual optimized by RDO for the current frame, and the predicted optimal motion vector and optimal prediction residual are no longer repeatedly calculated when encoding the frame structure data of the current frame. The optimal inter-frame prediction motion vector and prediction residual of the corresponding image frame in the reference bitstream are directly used for inter-frame prediction processing.

[0172] During the transformation process, the optimal transformation division method of the current frame can be obtained by decoding the reference bit stream after RDO optimization, and when encoding the frame structure data of the current frame, the optimal transformation division method is no longer repeatedly calculated, and the optimal transformation division method in the reference bit stream is directly used for transformation processing.

[0173] During the quantization process, the optimal quantization value of the quantizer optimized by RDO for the current frame can also be obtained by decoding the reference bit stream, and when encoding the frame structure data of the current frame, the optimal quantization value is no longer repeatedly calculated, and the optimal quantization value in the reference bit stream is directly used for quantization processing.

[0174] Afterwards, the reference bitstream may be decoded and entropy coding (lossless coding according to the information entropy principle) may be performed when encoding the frame structure data of the current frame, thereby obtaining the encoded image data of the current frame.

[0175] Under the encoding round-robin scheduling mechanism, N frames can be encoded in parallel at the same time to improve the processing efficiency of the bitstream data, and different frames can be allocated to different CPUs for encoding. For example, the CPU resources can be called to the maximum extent in a load balancing mode. If the image data processing device has two CPUs, the first frame is stored in the frame structure buffer 1 + reference bitstream buffer 1, and the encoding output bitstream is sent to the encoding output buffer 1 (the encoding output buffer is unified as a buffer pool. If the encoding output buffer 1 is occupied, the subsequent encoding output buffers 2..N are detected to see if there are free buffers, and the free encoding output buffers are output), and the encoding process runs on CPU1; the second frame is stored in the frame structure buffer 2 + reference bitstream buffer 2, and the encoding output bitstream is sent to the encoding output buffer 2, and the encoding process runs on CPU2; the third frame is stored in the frame structure buffer 3 + reference bitstream buffer 3, and the encoding output bitstream is sent to the encoding output buffer 3, and the encoding process runs on CPU1; and so on.

[0176] When the processing speeds of multiple CPUs are different, multiple code streams can be assigned to different CPUs for processing according to specific application scenarios. For example, the processing performance of each of the 1st to 4th CPUs is 3Mbps, and the processing performance of each of the 5th to 8th CPUs is 1Mbps. When processing multiple code streams, in order to maximize the use of CPU performance, for example, a total of 8 code streams, one solution is: first calculate the processing performance M1~M8 required for each code stream, traverse and calculate the sum of the single-channel processing performance value or the combined multi-channel processing performance value, and then compare it with 4Mbps to obtain the optimal solution for multiple code stream allocation.

[0177] (1) If the value of the required processing performance M3 of the third channel is less than or equal to 4Mbps and the value is closest to 4Mbps, the third channel code stream is allocated to the 5th to 8th CPUs for encoding processing in a load balancing mode, and the other 7 channels of code streams are allocated to the 1st to 4th CPUs for encoding processing in a load balancing mode;

[0178] (2) For example, if the sum of the processing performance required for the 5th and 7th channels, M5+M7, is less than or equal to 4Mbps and the value is closest to 4Mbps, then the 5th and 7th channels are allocated to the 5th to 8th CPUs for encoding processing in a load-balanced mode, and the other 6 channels are allocated to the 1st to 4th CPUs for encoding processing in a load-balanced mode.

[0179] The image data processing method provided by the present application is a technical solution for encoding round-robin scheduling. The decoding and encoding control implemented by round-robin scheduling can be used in software-implemented decoding and encoding solutions (encoding can be software or hardware). Through the round-robin scheduling buffer, balanced control improves CPU utilization, reuses reference encoding mode information, and improves the encoding processing speed of code stream data.

[0180] The present application also provides an image data processing device, such as Figure 4 As shown, including:

[0181] Memory 401, used for storing computer programs;

[0182] The processor 402 is used to implement any of the above-mentioned image data processing methods when executing the program stored in the memory 401.

[0183] Furthermore, the above-mentioned image data processing device may further include a communication bus and / or a communication interface, and the processor 402, the communication interface, and the memory 401 communicate with each other via the communication bus.

[0184] The communication bus mentioned in the above-mentioned image data processing device may be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus may be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0185] The communication interface is used for communication between the above-mentioned image data processing device and other devices.

[0186] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0187] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0188] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned image data processing methods are implemented.

[0189] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any one of the image data processing methods in the above embodiments.

[0190] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium. For example, the computer instructions may be transmitted from a website site, computer, server or data center to another website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a solid-state hard disk (SSD), etc.

[0191] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0192] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device embodiment, since it is basically similar to the method embodiment, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiment.

[0193] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for processing image data, characterized in that: Applied to an image data processing device, the method comprises: Obtaining decoded data of each frame obtained by decoding the target code stream; The decoded data of each frame is subjected to encoding preprocessing to obtain the frame structure data of each frame; wherein the frame structure data of any frame is data of a specified data format, and the storage form is the storage form of the frame structure; the specified data format is the data format required for encoding; wherein the encoding preprocessing of the decoded data of each frame to obtain the frame structure data of each frame includes: if the decoded data of each frame is the same as the specified data format, for any frame, determining the information about the frame structure of the decoded data of the frame, writing the determined frame structure information and a pointer for indicating the address of the decoded buffer of the frame into a target memory area, and using the target memory area as a frame structure buffer to map the decoded buffer of the frame into a frame structure buffer; wherein the information about the frame structure of the decoded data of the frame is: the structure information required to be used when the decoded data of the frame is subjected to encoding processing; the decoded buffer of the frame is used to store the decoded data of the frame; the frame structure data stored in each frame structure buffer is determined as the frame structure data of each frame; Perform auxiliary coding processing; wherein the auxiliary coding processing is: by calling the target encoder module, the frame structure data of each frame is encoded according to the reference code stream format to obtain the reference code stream processing; the coding mode information used when any frame is encoded into the reference code stream format is the same as the coding mode information used when the frame is encoded into the target code stream format; Determine the first coding mode information corresponding to each frame, and obtain reference coding mode information of each frame; wherein the first coding mode information corresponding to any frame is the coding mode information of the image frame corresponding to the frame in the reference bitstream; According to the acquired reference coding mode information, the decoded data of each frame is encoded to obtain the image data of each frame in the target bitstream format; The method further includes: in response to receiving an instruction to increase the image data processing speed, adjusting parameter information of a specified parameter used by a target encoder module when encoding; wherein the specified parameter is a parameter related to encoding complexity.

2. The method according to claim 1, characterized in that The step of encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame in the target bitstream format after encoding includes: According to the acquired reference coding mode information, the frame structure data of each frame is encoded to obtain the image data of each frame in the target code stream format after encoding.

3. The method according to claim 1, characterized in that A method for determining the first coding mode information corresponding to any frame includes: Acquire, from the coding mode information about each image frame of the reference bitstream, the coding mode information of the image frame corresponding to the frame, and obtain the first coding mode information corresponding to the frame; The method for determining the coding mode information of each image frame of the reference bitstream includes: Decoding the image data of each image frame in the reference bitstream to obtain the coding mode information about each image frame of the reference bitstream; or receiving the coding mode information of each image frame determined in the coding process of the reference bitstream transmitted by the target encoder module.

4. The method according to any one of claims 1 to 3, characterized in that: The method further comprises: Identify whether the target code stream format and the original code stream format meet a matching condition; wherein the matching condition includes: the encoding mode information used when any frame is encoded into the original code stream format is the same as the encoding mode information used when the frame is encoded into the target code stream format; the original code stream format is the code stream format of the target code stream; If the matching condition is not met, executing the step of determining the first coding mode information corresponding to each frame to obtain reference coding mode information of each frame; If the matching condition is met, the second coding mode information corresponding to each frame is determined, the reference coding mode information of each frame is obtained, and the step of encoding the decoded data of each frame according to the obtained reference coding mode information is performed to obtain the image data of each frame after encoding in the target code stream format; wherein the second coding mode information corresponding to any frame is obtained after decoding the image data of the frame.

5. The method according to any one of claims 1 to 3, characterized in that: The step of encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame in the target bitstream format after encoding includes: According to the prediction information in the obtained reference coding mode information, intra-frame prediction processing and inter-frame prediction processing are performed on the decoded data of each frame; According to the transformation information in the obtained reference coding mode information, transforming the data after the intra-frame prediction processing and the inter-frame prediction processing; quantizing the transformed data according to the quantization information in the obtained reference coding mode information; According to the entropy coding information in the obtained reference coding mode information, entropy coding is performed on the quantized data.

6. The method according to claim 1, characterized in that The encoding preprocessing is performed on the decoded data of each frame to obtain the frame structure data of each frame, and further includes: If the decoded data of each frame is different from the specified data format, for any frame, the decoded data of the frame is converted into a format according to the specified data format, and the converted data is stored in a frame structure buffer in an idle state in the frame structure buffer pool in the form of a frame structure; wherein different frame structure buffers store frame structure data of different frames, and each frame structure buffer is used to store frame structure data of one frame; The frame structure data stored in each frame structure buffer is determined as the frame structure data of each frame.

7. The method according to claim 3, characterized in that The decoding process of the image data of each image frame in the reference bitstream to obtain the coding mode information of each image frame in the reference bitstream includes: Decoding the image data of the image frames in each reference code stream buffer to obtain coding mode information about each image frame of the reference code stream; wherein different reference code stream buffers store image data of different image frames in the reference code stream, and each reference code stream buffer is used to store image data of one image frame in the reference code stream.

8. The method according to any one of claims 1 to 3, characterized in that: The step of encoding the decoded data of each frame according to the acquired reference coding mode information to obtain the image data of each frame in the target bitstream format after encoding includes: The decoded data of each frame is distributed to the target CPU in frame order, so that the target CPU calls the specified encoder module, encodes the decoded data of each frame according to the obtained reference encoding mode information, and obtains the image data of each frame after encoding in the target code stream format.

9. An image data processing device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-8 when executing a program stored in a memory.

Images