Decoding method compatible with multiple code streams

Abstract

The invention discloses a decoding method compatible with multiple code streams. The decoding method comprises the following steps: S1, transplanting ffmpeg to an acceleration module platform; s2, setting an initialization resource calling interface; s3, using an ffmpeg dynamic library to obtain a complete compressed image data packet containing a parameter frame; s4, screening and filtering the complete code stream; s5, performing hard decoding on the screened and filtered code stream; s6, setting an error callback function to check a decoding state; s7, returning the decoded image to the upper application through an interface function; and S8, releasing decoding channel resources. A soft decoding method is adopted to form a complete data packet containing an image frame and a parameter frame, and then a hard decoding module of a chip is used to complete the decoding process, so that the decoding time is effectively shortened.

Description

Technical Field

[0001] The present invention belongs to the technical field of video decoding under Linux system. In the application scenario where the system is Linux, according to the characteristics of atlas200AI chip, combined with hardware properties and soft decoding characteristics, a video decoding method compatible with multiple code streams under Linux system is designed and implemented. The present invention is that the system is Linux, firstly, the FFMPEG soft decoding library is used to analyze the data received by the network, and the compressed frame parameter information and image information are effectively obtained. Different screening mechanisms are established according to the coding characteristics of the code stream, and the decodable data packets are transmitted to the atlas200AI chip, thereby completing the decoding process. Background Art

[0002] The Atlas 200AI acceleration module integrates the Ascend 310AI processor, which can realize image recognition, image classification, etc. on the edge. It can realize various data analysis and reasoning calculations such as images and videos, and is widely used in edge AI scenarios such as smart cameras, robots, and drones.

[0003] However, Atlas 200AI belongs to the category of hard decoding. When the protocol frame does not fully comply with the decoding protocol, or there are many error frames, the decoding efficiency is relatively low or cannot be decoded, and for image information with parameter frames, the hard decoding module cannot extract parameter information. The present invention uses the ffmpeg decoding library to obtain the parameter frame information and data frame information in the original frame, obtains the complete image frame data packet, transmits the decodable complete data packet to the hard decoding module, completes the decoding function, and at the same time, in order to solve the call interface for the upper-layer application, it is provided to the upper layer in the form of a dynamic library. In addition, in order to avoid the hardware from being unable to release memory resources due to decoding timeout caused by error frames, monitoring means are used to release memory in time to avoid memory leakage. Summary of the invention

[0004] (I) Purpose of the invention

[0005] The purpose of the present invention is to adopt a software-hardware combined decoding method based on an acceleration module to provide a video decoding method compatible with multiple code streams and solve the video decoding problem under the Linux system.

[0006] (II) Technical solution

[0007] In order to solve the above technical problem, the present invention provides a decoding method compatible with multiple code streams, which comprises the following steps:

[0008] S1: port ffmpeg to the acceleration module platform;

[0009] S2: Set up the initialization resource calling interface;

[0010] S3: Use the ffmpeg dynamic library to obtain the complete compressed image data packet containing the parameter frame;

[0011] S4: Filter the complete bitstream;

[0012] S5: hard decoding the filtered bit stream;

[0013] S6: Set the error callback function to check the decoding status;

[0014] S7: Return the decoded image to the upper layer application through the interface function;

[0015] S8: Release the decoding channel resources.

[0016] Step S1 includes the following process:

[0017] S11: Configure ffmpeg compilation properties, and configure parameters according to platform type, cpu type, codec properties, format conversion properties, and cross-compilation properties;

[0018] S12: Modify the acodec.h file, add the parameter frame length sei_len and array sei_buf, and determine its parameter frame size SEI_BUF_SIZE according to the application layer requirements;

[0019] S13: Add the function of obtaining the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: obtain the parameter size in the function, assign size to sei_len, determine whether size is less than or equal to SEI_BUF_SIZE, and if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array subscript i of the copied sei_buf is the index index of the gb array divided by 8, that is, sei_buf[i]=gb_buf[index / 8];

[0020] S14: Execute the configure command to generate decoding libraries libavcodec, libavformat, libavutil, and libswscale in the subfolder lib of the configuration folder.

[0021] S15: Copy the dynamic library to the / usr / lib path of the decoding board.

[0022] In step S2, the calling interface is set as an external function, the parameters and return values ​​are set according to the protocol, and the decoding resources are initialized. The process is as follows:

[0023] S21: Initialize semaphore mutex;

[0024] S22: Create an initialization resource setting thread. The process of setting the thread main logic content is as follows:

[0025] S221: Setting the hardware decoding device serial number;

[0026] S222: Use library functions to create a hard decoding context environment;

[0027] S223: Use library functions to create a hard-decoded data stream object;

[0028] S224: Create a decoding thread;

[0029] S225: Initialize ffmpeg resources;

[0030] S226: Create a network data receiving thread and put the received data into a ring buffer array;

[0031] S227: Initialize the accelerated decoding module resources. The process is as follows:

[0032] S2271: Create decoding channel attributes. If the creation is successful, proceed to the next step, otherwise exit with an error;

[0033] S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step, otherwise an error is reported and exit;

[0034] S2273: Set the decoding thread number;

[0035] S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0036] S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step, otherwise report an error and exit;

[0037] S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0038] S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0039] S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0040] S2279: Create a decoding channel according to the decoding channel attributes;

[0041] S228: Periodically query whether the network has received the image array, that is, whether the ring buffer array has data. If so, take the data according to the frame length specified by the protocol and proceed to the next process. Otherwise, continue to query and wait for network data.

[0042] Step S3 includes the following process:

[0043] S31: Initialize the decoding library usage environment

[0044] First, set the decoder type and use the HEVC / h265 type decoder; then initialize the decoder context and dynamically apply for image frame storage space;

[0045] S32: Create ffmpeg to obtain data packets thread, the process is as follows:

[0046] S321: Get the first address pointer bufPtr and length bufLen of the image data array read from the network;

[0047] S322: Determine whether the current data length bufLen is greater than 0, if so, proceed to the next step, if not, wait for the network to send data next time;

[0048] S323: Send the data array first address pointer bufPtr and length bufLen to the soft decoding module, and use the library function av_parser_parse2 to split the data into frames. If a complete image frame data packet can be successfully obtained from the array, it will be recorded and continue to the next step. If not, the decoding process will be exited.

[0049] S324: Save the length ret of the data packet of the image frame segmentation in the data array this time, remove the length ret of the data packet of the image frame segmentation this time from the total length bufLen of the data array, and move the first address pointer bufPtr forward by the number of times ret;

[0050] S325: Put the complete data packet of the image segmented this time into a queue to be decoded.

[0051] Step S4 includes the following process:

[0052] S41: Setting the upper layer application to call the interface to obtain the code stream encoding type;

[0053] S42: Determine whether the encoding type is a code stream without an I frame, and then start mode 1 for screening. The screening process of mode 1 is as follows:

[0054] S421: Check whether the frame type of the current channel is consistent with the frame type previously saved. If yes, proceed to the next step; otherwise, jump to S4224;

[0055] S422: setting the identifier indertIDR of inserting an I frame to 1;

[0056] S423: Determine whether the type value of the current frame is 0xff, or 0x00, or 0xaa, or 0xf0;

[0057] S424: If yes, save the type of the current frame to type as the latest frame type mark, otherwise exit the screening process;

[0058] S425: Get the length of the current frame and save it to the insertion frame length idrsize;

[0059] S426: traverse the data of the current frame, frame position i from the first data to idrsize-3 data;

[0060] S427: Determine whether the data at the current frame position i is 0x00, the second data i+1 is 0x00, and the third data i+2 is 0x01. If yes, proceed to the next step; otherwise, jump to S426 to continue traversal;

[0061] S428: Determine whether the data at the current frame position i+3 is 0x40, or equal to 0x42, or equal to 0x44. If yes, proceed to the next step; otherwise, jump to S4212

[0062] S429: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4211;

[0063] S4210: Set the flag insertIDR of inserting I frame to 1, and exit the screening process;

[0064] S4211: Set the I frame flag idrFalg to 1;

[0065] S4212: Determine whether the data at the current frame position i+3 is 0x4E, if yes, proceed to the next step, otherwise jump to S4216;

[0066] S4213: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4215;

[0067] S4214: Set the insert I frame flag insertIDR to 1, and exit the screening process;

[0068] S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start;

[0069] S4216: Determine whether the data at the current frame position i+3 is 0x26, if yes, proceed to the next step, otherwise jump to S4220;

[0070] S4217: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4219;

[0071] S4218: Set the insert I frame flag insertIDR to 1, and exit the screening process;

[0072] S4219: record the number of P frames of the current frame, isliceNumber, increase by 1, and set the number of frames to 0;

[0073] S4220: Change the current frame type to 0x02, that is, set the data at position i+3 to 0x02;

[0074] S4221: Determine whether the data at the current frame position i+3 is 0x02. If yes, continue to determine whether the parameter frame flag seiFlag is 0. If yes, set the insert I frame flag insertIDR to 1, and end the screening process and wait for the next message to be received;

[0075] S4222: Determine whether the indication of inserting an I frame is 1. If so, perform framing according to the saved frame type type. During framing, determine the number of current P frames isliceNumber and fill in the framing content according to the different numbers. Otherwise, proceed to the next step.

[0076] S4223: modifying the frame content according to the saved frame type type, and modifying the position of the current frame differently according to the number of the current P frame isliceNumber;

[0077] S4224: If the current frame is not a parameter frame or there is no I frame or no P frame, exit the process;

[0078] S4225: dynamically apply for the memory of the image frame, and copy the parameter information of the frame and the I frame information in the group as needed according to the screening process and the frame grouping process in the above steps. If the frame needs to be inserted with an I frame, skip this process and go directly to the next step;

[0079] S4226: Add the filtered frame content to the requested image frame memory array;

[0080] S43: Determine whether the encoding type is a standard code stream containing an I frame, and then start mode 2 for screening. The screening process of mode 2 is as follows:

[0081] S431: dynamically apply for memory space inbuffer for image frames;

[0082] S432: copy the size of the current frame to store it in the array size to be decoded inbuffersize;

[0083] S433: Use the library function acltMemcpy to copy the data of the frame to inbuffer, and the size of the copy is size;

[0084] S44: the number of parameter frames seiCount recording the data structure to be decoded is increased by 1;

[0085] S45: Determine whether the current size of seiCount is greater than the maximum number of macro-defined parameter frames MAX_SEI_NUM, if so, clear the current number of parameter frames seiCount, otherwise proceed to the next step;

[0086] S46: Save the size of the current parameter frame seiSize to the data structure to be decoded, and copy the content of the parameter frame to the parameter array seiData of the data structure to be decoded.

[0087] In step S5, the complete data packet filtered by ffmpeg is hard-decoded, and the hard decoding includes the following process:

[0088] S51: Set the decoding callback function. The process is as follows:

[0089] S511: Use the library function to obtain the normal frame output of the decoding. If the obtained result is 0, execute the next step, otherwise report an error and exit;

[0090] S512: using library functions to obtain image attribute data output;

[0091] S513: Use library functions to obtain image size, width and height parameter data from output;

[0092] S514: copy the image data and parameter data to the user return array g_send_buf;

[0093] S515: Unlock the semaphore mutex and allow access to the user return array g_send_buf;

[0094] S52: Obtaining a complete image data packet from step S4;

[0095] S53: Send the data packet to the hard decoding module for decoding. The decoding process is as follows:

[0096] S531: Use library functions to dynamically apply for image storage space;

[0097] S532: Create an input image stream attribute object;

[0098] S533: Set the input image stream attribute object;

[0099] S534: Dynamically apply for image output memory space;

[0100] S535: Create an output image attribute object;

[0101] S536: Setting output image attributes according to image data, size, and format;

[0102] S537: Using the decoding channel attribute, input image stream attribute, and output image attribute as parameters, the library function is used to send the image frame data to the hard decoding module;

[0103] S538: Release the requested image memory space;

[0104] S539: Destroy the input image stream object resources.

[0105] In step S6, during the decoding process, it is checked in real time whether the decoded image frame can be decoded smoothly. If not, resources need to be released. The process is as follows:

[0106] S61: Set the error callback function to callbackErr, and the parameter is the exception type object pointer;

[0107] S62: using a library function to obtain the decoding task number taskid in the abnormal situation information;

[0108] S63: Obtain the image frame number frameid of the decoded stream in the abnormal situation information;

[0109] S64: Obtaining the device number in the abnormal situation decoding device;

[0110] S65: Check whether the device id and task taskid in the previous initialization settings are consistent. If yes, use the memory space applied for in step S5 by the image frame number frameid.

[0111] In step S7, after obtaining the hard-decoded image, the image data is returned through the interface function, and the process is as follows:

[0112] S71: Set the readbuf function to provide an interface for obtaining image data. The content is as follows:

[0113] S711: The semaphore is locked to prevent read and write conflicts in the user return array;

[0114] S712: copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf;

[0115] S713: Return the image data length;

[0116] S72: Set and provide a function void*getbufAddress for obtaining the address of the image data array, which returns the address of g_read_buf.

[0117] The process of step S8 is as follows:

[0118] S81: calling the library function to report the decoding thread ID and data stream ID described;

[0119] S82: Set the flag of the decoding thread to false;

[0120] S83: Release the decoding thread;

[0121] S84: Destroy the decoding channel object;

[0122] S85: Set the decoding channel description pointer to null;

[0123] S86: Destroy the decoding stream description object;

[0124] S87: Destroy the decoding stream object;

[0125] S88: Set the array object in the decoded stream to empty.

[0126] (III) Beneficial effects

[0127] The multi-stream compatible video decoding method provided by the above technical solution can solve the video decoding and transmission problems under the Linux system. The method has been verified by the algorithm and experimentally tested. The results show that the solution can solve the image decoding problem by using a soft decoding method to form a complete data packet containing image frames and parameter frames, and then use the chip's hard decoding module to complete the decoding process, effectively reducing the decoding time. In addition, a screening mechanism is established to enable different decoding mechanisms for different streams to improve decoding efficiency. DETAILED DESCRIPTION

[0128] In order to make the purpose, content and advantages of the present invention more clear, the specific implementation methods of the present invention are further described in detail below in conjunction with embodiments.

[0129] The video decoding method compatible with multiple code streams in this embodiment includes the following steps:

[0130] S1: Porting ffmpeg to the acceleration module platform

[0131] S11: Configure ffmpeg compilation properties, and configure parameters according to platform type, CPU type, codec properties, format conversion properties, and cross-compilation properties.

[0132] S12: Modify the acodec.h file, add the parameter frame length sei_len and array sei_buf, and determine its parameter frame size SEI_BUF_SIZE according to the application layer requirements;

[0133] S13: Add the function of obtaining the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: obtain the parameter size in the function, assign size to sei_len, determine whether size is less than or equal to SEI_BUF_SIZE, and if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array subscript i of the copied sei_buf is the index index of the gb array divided by 8, that is, sei_buf[i]=gb_buf[index / 8];

[0134] S14: Execute the configure command to generate decoding libraries libavcodec, libavformat, libavutil, and libswscale in the subfolder lib of the configuration folder.

[0135] S15: Copy the dynamic library to the / usr / lib path of the decoding board;

[0136] S2: Set up the initialization resource call interface

[0137] Set the calling interface to an external function, set the parameters and return values ​​according to the protocol, and initialize the decoding resources. The specific operations are as follows:

[0138] S21: Initialize semaphore mutex;

[0139] S22: Create an initialization resource setting thread. The main logic content of the thread is as follows:

[0140] S221: Setting the hardware decoding device serial number;

[0141] S222: Use library functions to create a hard decoding context environment;

[0142] S223: Use library functions to create a hard-decoded data stream object;

[0143] S224: Create a decoding thread;

[0144] S225: Initialize ffmpeg resources;

[0145] S226: Create a network data receiving thread and put the received data into a ring buffer array;

[0146] S227: Initialize the accelerated decoding module resources. The specific operations are as follows:

[0147] S2271: Create decoding channel attributes. If the creation is successful, proceed to the next step, otherwise exit with an error;

[0148] S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step, otherwise an error is reported and exit;

[0149] S2273: Set the decoding thread number;

[0150] S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0151] S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step, otherwise report an error and exit;

[0152] S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0153] S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0154] S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step, otherwise exit with an error;

[0155] S2279: Create a decoding channel according to the decoding channel attributes;

[0156] S228: Regularly query whether the network has received the image array, that is, whether the ring buffer array has data. If yes, take the data according to the frame length specified by the protocol and proceed to the next process. Otherwise, continue to query and wait for network data.

[0157] S3: Use ffmpeg dynamic library to obtain complete compressed image data packets including parameter frames

[0158] S31: Initialize the decoding library usage environment

[0159] First, set the decoder type and use the HEVC (h265) decoder. Then, initialize the decoder context and dynamically apply for image frame storage space.

[0160] S32: Create ffmpeg to obtain data packets thread, the specific process of the thread is as follows:

[0161] S321: Get the first address pointer bufPtr and length bufLen of the image data array read from the network;

[0162] S322: Determine whether the current data length bufLen is greater than 0, if so, proceed to the next step, if not, wait for the network to send data next time.

[0163] S323: Send the data array first address pointer bufPtr and length bufLen to the soft decoding module, and use the library function av_parser_parse2 to split the data into frames. If a complete image frame data packet can be successfully obtained from the array, record it and continue to the next step. If not, exit the decoding process.

[0164] S324: Save the data packet length ret of the image frame segmentation in this data array, remove the data packet length ret of the image frame segmentation this time from the total length bufLen of the data array, and move the first address pointer bufPtr forward by the number of moves ret.

[0165] S325: Put the complete data packet of the image segmented this time into a queue to be decoded.

[0166] S4: Filter the complete bitstream

[0167] The specific steps are as follows:

[0168] S41: Setting the upper layer application to call the interface to obtain the code stream encoding type;

[0169] S42: Determine whether the encoding type is a code stream without an I frame, and then start mode 1 for screening. The specific steps are as follows:

[0170] S421: Check whether the frame type of the current channel is consistent with the frame type previously saved. If yes, proceed to the next step; otherwise, jump to S4224;

[0171] S422: setting the identifier indertIDR of inserting an I frame to 1;

[0172] S423: Determine whether the type value of the current frame is 0xff, or 0x00, or 0xaa, or 0xf0;

[0173] S424: If yes, save the type of the current frame to type as the latest frame type mark, otherwise exit the screening process;

[0174] S425: Get the length of the current frame and save it to the insertion frame length idrsize;

[0175] S426: traverse the data of the current frame, frame position i from the first data to idrsize-3 data;

[0176] S427: Determine whether the data at the current frame position i is 0x00, the second data i+1 is 0x00, and the third data i+2 is 0x01. If yes, proceed to the next step; otherwise, jump to S426 to continue traversal;

[0177] S428: Determine whether the data at the current frame position i+3 is 0x40, or equal to 0x42, or equal to 0x44. If yes, proceed to the next step; otherwise, jump to S4212

[0178] S429: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4211;

[0179] S4210: Set the flag insertIDR of inserting I frame to 1, and exit the screening process;

[0180] S4211: Set the I frame flag idrFalg to 1;

[0181] S4212: Determine whether the data at the current frame position i+3 is 0x4E, if yes, proceed to the next step, otherwise jump to S4216;

[0182] S4213: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4215;

[0183] S4214: Set the insert I frame flag insertIDR to 1, and exit the screening process;

[0184] S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start;

[0185] S4216: Determine whether the data at the current frame position i+3 is 0x26, if yes, proceed to the next step, otherwise jump to S4220;

[0186] S4217: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4219;

[0187] S4218: Set the insert I frame flag insertIDR to 1, and exit the screening process;

[0188] S4219: record the number of P frames of the current frame, isliceNumber, increase by 1, and set the number of frames to 0;

[0189] S4220: Change the current frame type to 0x02, that is, set the data at position i+3 to 0x02;

[0190] S4221: Determine whether the data at the current frame position i+3 is 0x02. If yes, continue to determine whether the parameter frame flag seiFlag is 0. If yes, set the insert I frame flag insertIDR to 1, and end the screening process and wait for the next message to be received;

[0191] S4222: Determine whether the indication of inserting an I frame is 1. If so, perform framing according to the saved frame type type. During framing, determine the number of current P frames isliceNumber and fill in the framing content according to the different numbers. Otherwise, proceed to the next step.

[0192] S4223: modifying the frame content according to the saved frame type type, and modifying the position of the current frame differently according to the number of the current P frame isliceNumber;

[0193] S4224: If the current frame is not a parameter frame or there is no I frame or no P frame, exit the process;

[0194] S4225: dynamically apply for the memory of the image frame, and copy the parameter information of the frame and the I frame information in the group as needed according to the screening process and the frame grouping process in the above steps. If the frame needs to be inserted with an I frame, skip this process and go directly to the next step;

[0195] S4226: Add the filtered frame content to the requested image frame memory array;

[0196] S43: Determine whether the encoding type is a standard bitstream containing an I frame, and then start mode 2 for screening. The specific process is as follows:

[0197] S431: dynamically apply for memory space inbuffer for image frames;

[0198] S432: copy the size of the current frame to store it in the array size to be decoded inbuffersize;

[0199] S433: Use the library function acltMemcpy to copy the data of the frame to inbuffer, and the size of the copy is size;

[0200] S44: the number of parameter frames seiCount recording the data structure to be decoded is increased by 1;

[0201] S45: Determine whether the current size of seiCount is greater than the maximum number of macro-defined parameter frames MAX_SEI_NUM, if so, clear the current number of parameter frames seiCount, otherwise proceed to the next step;

[0202] S46: Save the size of the current parameter frame seiSize to the data structure to be decoded, and copy the content of the parameter frame to the parameter array seiData of the data structure to be decoded;

[0203] S5: Hard decoding of the filtered bitstream

[0204] Hard decode the complete data packet after ffmpeg filtering. The specific steps are as follows:

[0205] S51: Set the decoding callback function, as follows:

[0206] S511: Use the library function to obtain the normal frame output of the decoding. If the obtained result is 0, execute the next step, otherwise report an error and exit;

[0207] S512: using library functions to obtain image attribute data output;

[0208] S513: Use library functions to obtain image size, width and height parameter data from output;

[0209] S514: copy the image data and parameter data to the user return array g_send_buf;

[0210] S515: Unlock the semaphore mutex and allow access to the user return array g_send_buf;

[0211] S52: Obtaining a complete image data packet from step S4;

[0212] S53: Send the data packet to the hard decoding module for decoding, as follows:

[0213] S531: Use library functions to dynamically apply for image storage space;

[0214] S532: Create an input image stream attribute object;

[0215] S533: Set the input image stream attribute object;

[0216] S534: Dynamically apply for image output memory space;

[0217] S535: Create an output image attribute object;

[0218] S536: Setting output image attributes according to image data, size, and format;

[0219] S537: Using the decoding channel attribute, input image stream attribute, and output image attribute as parameters, the library function is used to send the image frame data to the hard decoding module;

[0220] S538: Release the requested image memory space;

[0221] S539: Destroy the input image stream object resource;

[0222] S6: Set the error callback function to check the decoding status

[0223] During the decoding process, check in real time whether the decoded image frame can be decoded smoothly. If not, you need to release the resources. The specific steps are as follows:

[0224] S61: Set the error callback function to callbackErr, and the parameter is the exception type object pointer;

[0225] S62: using a library function to obtain the decoding task number taskid in the abnormal situation information;

[0226] S63: Obtain the image frame number frameid of the decoded stream in the abnormal situation information;

[0227] S64: Obtaining the device number in the abnormal situation decoding device;

[0228] S65: Check whether the device id and task taskid in the previous initialization setting are consistent. If yes, use the memory space applied for in step S5 by the image frame number frameid;

[0229] S7: Return the decoded image to the upper layer application through the interface function

[0230] After obtaining the hard-decoded image, the image data is returned through the interface function. The specific steps are as follows:

[0231] S71: Set the readbuf function to provide an interface for obtaining image data. The specific contents are as follows:

[0232] S711: The semaphore is locked to prevent read and write conflicts in the user return array;

[0233] S712: copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf;

[0234] S713: Return the image data length;

[0235] S72: Set the function void*getbufAddress to obtain the address of the image data array. This function returns the address of g_read_buf:

[0236] S8: Release the decoding channel resources

[0237] The specific steps are as follows:

[0238] S81: calling the library function to report the decoding thread ID and data stream ID described;

[0239] S82: Set the flag of the decoding thread to false (no decoding);

[0240] S83: Release the decoding thread;

[0241] S84: Destroy the decoding channel object;

[0242] S85: Set the decoding channel description pointer to null;

[0243] S86: Destroy the decoding stream description object;

[0244] S87: Destroy the decoding stream object;

[0245] S88: Set the array object in the decoded stream to empty.

[0246] The above is only a preferred embodiment of the present invention. It should be pointed out that for ordinary technicians in this technical field, several improvements and modifications can be made without departing from the technical principles of the present invention. These improvements and modifications should also be regarded as the scope of protection of the present invention.

Claims

1. A decoding method compatible with multiple code streams, characterized in that: The following steps are involved: S1: port ffmpeg to the acceleration module platform; S2: Set up the initialization resource calling interface; S3: Use the ffmpeg dynamic library to obtain the complete compressed image data packet containing the parameter frame; S4: Filter the complete bitstream; S5: hard decoding the filtered bit stream; S6: Set the error callback function to check the decoding status; S7: Return the decoded image to the upper layer application through the interface function; S8: Release the decoding channel resources.

2. The multi-stream compatible decoding method according to claim 1, characterized in that: Step S1 includes the following process: S11: Configure ffmpeg compilation properties, and configure parameters according to platform type, cpu type, codec properties, format conversion properties, and cross-compilation properties; S12: Modify the acodec.h file, add the parameter frame length sei_len and array sei_buf, and determine its parameter frame size SEI_BUF_SIZE according to the application layer requirements; S13: Add the function of obtaining the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: obtain the parameter size in the function, assign size to sei_len, determine whether size is less than or equal to SEI_BUF_SIZE, and if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array subscript i of the copied sei_buf is the index index of the gb array divided by 8, that is, sei_buf[i]=gb_buf[index / 8]; S14: Execute the configure command to generate decoding libraries libavcodec, libavformat, libavutil, and libswscale in the subfolder lib of the configuration folder. S15: Copy the dynamic library to the / usr / lib path of the decoding board.

3. The multi-stream compatible decoding method according to claim 2, wherein: In step S2, the calling interface is set as an external function, the parameters and return values ​​are set according to the protocol, and the decoding resources are initialized. The process is as follows: S21: Initialize semaphore mutex; S22: Create an initialization resource setting thread. The process of setting the thread main logic content is as follows: S221: Setting the hardware decoding device serial number; S222: Use library functions to create a hard decoding context environment; S223: Use library functions to create a hard-decoded data stream object; S224: Create a decoding thread; S225: Initialize ffmpeg resources; S226: Create a network data receiving thread and put the received data into a ring buffer array; S227: Initialize the accelerated decoding module resources. The process is as follows: S2271: Create decoding channel attributes. If the creation is successful, proceed to the next step, otherwise exit with an error; S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step, otherwise an error is reported and exit; S2273: Set the decoding thread number; S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step, otherwise exit with an error; S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step, otherwise report an error and exit; S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error; S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step, otherwise exit with an error; S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step, otherwise exit with an error; S2279: Create a decoding channel according to the decoding channel attributes; S228: Periodically query whether the network has received the image array, that is, whether the ring buffer array has data. If so, take the data according to the frame length specified by the protocol and proceed to the next process. Otherwise, continue to query and wait for network data.

4. The multi-stream compatible decoding method according to claim 3, characterized in that: Step S3 The process includes: S31: Initialize the decoding library usage environment First, set the decoder type and use the HEVC / h265 type decoder; then initialize the decoder context and dynamically apply for image frame storage space; S32: Create ffmpeg to obtain data packets thread, the process is as follows: S321: Get the first address pointer bufPtr and length bufLen of the image data array read from the network; S322: Determine whether the current data length bufLen is greater than 0, if so, proceed to the next step, if not, wait for the network to send data next time; S323: Send the data array first address pointer bufPtr and length bufLen to the soft decoding module, and use the library function av_parser_parse2 to split the data into frames. If a complete image frame data packet can be successfully obtained from the array, it will be recorded and continue to the next step. If not, the decoding process will be exited. S324: Save the length ret of the data packet of the image frame segmentation in the data array this time, remove the length ret of the data packet of the image frame segmentation this time from the total length bufLen of the data array, and move the first address pointer bufPtr forward by the number of times ret; S325: Put the complete data packet of the image segmented this time into a queue to be decoded.

5. The multi-stream compatible decoding method according to claim 4, characterized in that: Step S4 The process includes: S41: Setting the upper layer application to call the interface to obtain the code stream encoding type; S42: Determine whether the encoding type is a code stream without an I frame, and then start mode 1 for screening. The screening process of mode 1 is as follows: S421: Check whether the frame type of the current channel is consistent with the frame type previously saved. If yes, proceed to the next step; otherwise, jump to S4224; S422: setting the identifier indertIDR of inserting an I frame to 1; S423: Determine whether the type value of the current frame is 0xff, or 0x00, or 0xaa, or 0xf0; S424: If yes, save the type of the current frame to type as the latest frame type mark, otherwise exit the screening process; S425: Get the length of the current frame and save it to the insertion frame length idrsize; S426: traverse the data of the current frame, frame position i from the first data to idrsize-3 data; S427: Determine whether the data at the current frame position i is 0x00, the second data i+1 is 0x00, and the third data i+2 is 0x01. If yes, proceed to the next step; otherwise, jump to S426 to continue traversal; S428: Determine whether the data at the current frame position i+3 is 0x40, or equal to 0x42, or equal to 0x44. If yes, proceed to the next step; otherwise, jump to S4212 S429: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4211; S4210: Set the flag insertIDR for inserting an I frame to 1, and exit the screening process; S4211: Set the I frame flag idrFalg to 1; S4212: Determine whether the data at the current frame position i+3 is 0x4E, if yes, proceed to the next step, otherwise jump to S4216; S4213: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4215; S4214: Set the insert I frame flag insertIDR to 1, and exit the screening process; S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start; S4216: Determine whether the data at the current frame position i+3 is 0x26, if yes, proceed to the next step, otherwise jump to S4220; S4217: Determine whether the parameter flag seiFlag is 1, if so, proceed to the next step, otherwise jump to S4219; S4218: Set the insert I frame flag insertIDR to 1, and exit the screening process; S4219: record the number of P frames of the current frame, isliceNumber, increase by 1, and set the number of frames to 0; S4220: Change the current frame type to 0x02, that is, set the data at position i+3 to 0x02; S4221: Determine whether the data at the current frame position i+3 is 0x02. If yes, continue to determine whether the parameter frame flag seiFlag is 0. If yes, set the insert I frame flag insertIDR to 1, and end the screening process and wait for the next message to be received; S4222: determining whether the indication of inserting an I frame is 1, and if so, performing framing according to the saved frame type type, and determining the number of current P frames isliceNumber while framing, and completing framing content according to different numbers; Otherwise proceed to the next step; S4223: modifying the frame content according to the saved frame type type, and modifying the position of the current frame differently according to the number of the current P frame isliceNumber; S4224: If the current frame is not a parameter frame or there is no I frame or no P frame, exit the process; S4225: dynamically apply for the memory of the image frame, and copy the parameter information of the frame and the I frame information in the group as needed according to the screening process and the frame grouping process in the above steps. If the frame needs to be inserted with an I frame, skip this process and go directly to the next step; S4226: Add the filtered frame content to the requested image frame memory array; S43: Determine whether the encoding type is a standard code stream containing an I frame, and then start mode 2 for screening. The screening process of mode 2 is as follows: S431: dynamically apply for memory space inbuffer for image frames; S432: copy the size of the current frame to store it in the array size to be decoded inbuffersize; S433: Use the library function acltMemcpy to copy the data of the frame to inbuffer, and the size of the copy is size; S44: the number of parameter frames seiCount recording the data structure to be decoded is increased by 1; S45: Determine whether the current size of seiCount is greater than the maximum number of macro-defined parameter frames MAX_SEI_NUM, if so, clear the current number of parameter frames seiCount, otherwise proceed to the next step; S46: Save the size of the current parameter frame seiSize to the data structure to be decoded, and copy the content of the parameter frame to the parameter array seiData of the data structure to be decoded.

6. The multi-stream compatible decoding method according to claim 5, characterized in that: In step S5, the complete data packet filtered by ffmpeg is hard-decoded, and the hard decoding includes the following process: S51: Set the decoding callback function. The process is as follows: S511: Use the library function to obtain the normal frame output of the decoding. If the obtained result is 0, execute the next step, otherwise report an error and exit; S512: using library functions to obtain image attribute data output; S513: Use library functions to obtain image size, width and height parameter data from output; S514: copy the image data and parameter data to the user return array g_send_buf; S515: Unlock the semaphore mutex and allow access to the user return array g_send_buf; S52: Obtaining a complete image data packet from step S4; S53: Send the data packet to the hard decoding module for decoding. The decoding process is as follows: S531: Use library functions to dynamically apply for image storage space; S532: Create an input image stream attribute object; S533: Set the input image stream attribute object; S534: Dynamically apply for image output memory space; S535: Create an output image attribute object; S536: Setting output image attributes according to image data, size, and format; S537: Using the decoding channel attribute, input image stream attribute, and output image attribute as parameters, the library function is used to send the image frame data to the hard decoding module; S538: Release the requested image memory space; S539: Destroy the input image stream object resources.

7. The multi-stream compatible decoding method according to claim 6, wherein: In step S6, during the decoding process, it is checked in real time whether the decoded image frame can be decoded smoothly. If not, resources need to be released. The process is as follows: S61: Set the error callback function to callbackErr, and the parameter is the exception type object pointer; S62: using a library function to obtain the decoding task number taskid in the abnormal situation information; S63: Obtain the image frame number frameid of the decoded stream in the abnormal situation information; S64: Obtaining the device number in the abnormal situation decoding device; S65: Check whether the device id and task taskid in the previous initialization settings are consistent. If yes, use the memory space applied for in step S5 by the image frame number frameid.

8. The multi-stream compatible decoding method according to claim 7, characterized in that: In step S7, after obtaining the hard-decoded image, the image data is returned through the interface function, and the process is as follows: S71: Set the readbuf function to provide an interface for obtaining image data. The content is as follows: S711: The semaphore is locked to prevent read and write conflicts in the user return array; S712: copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf; S713: Return the image data length; S72: Set and provide a function void*getbufAddress for obtaining the address of the image data array, which returns the address of g_read_buf.

9. The multi-stream compatible decoding method according to claim 8, characterized in that: The process of step S8 is as follows: S81: calling the library function to report the decoding thread ID and data stream ID described; S82: Set the flag of the decoding thread to false; S83: Release the decoding thread; S84: Destroy the decoding channel object; S85: Set the decoding channel description pointer to null; S86: Destroy the decoding stream description object; S87: Destroy the decoding stream object; S88: Set the array object in the decoded stream to empty.

10. An application of the multi-stream compatible decoding method according to any one of claims 1 to 9 in the field of video decoding technology under Linux system.