A decoding method compatible with multiple bitstreams

Abstract

The application discloses a decoding method compatible with multiple code streams, and steps are as follows: S1, transplanting ffmpeg to an acceleration module platform; S2, setting an initialization resource calling interface; S3, using an ffmpeg dynamic library to acquire a complete compressed image data packet containing a parameter frame; S4, filtering the complete code stream; S5, hard decoding the filtered code stream; S6, setting an error callback function to check a decoding state; S7, returning the decoded image to an upper layer application through an interface function; and S8, releasing a decoding channel resource. The application adopts a soft decoding method, forms a complete data packet containing an image frame and a parameter frame, and then uses a chip hard decoding module to complete a decoding process, so that the decoding time is effectively reduced.

Description

Technical Field

[0001] This invention belongs to the field of video decoding technology under Linux systems. In Linux-based application scenarios, considering the characteristics of the Atlas200AI chip and combining hardware attributes and software decoding features, a multi-stream compatible video decoding method for Linux systems is designed and implemented. This invention, for Linux systems, first uses the FFMPEG software decoding library to analyze the data received from the network, effectively obtaining compressed frame parameter information and image information. Then, it analyzes the encoding characteristics of the bitstreams to establish different filtering mechanisms, and transmits decodable data packets to the Atlas200AI chip, thereby completing the decoding process. Background Technology

[0002] The Atlas 200 AI acceleration module integrates the Ascend 310 AI processor, enabling image recognition, image classification, and other functions on the edge. It can perform various data analysis and inference calculations, including images and videos, and is widely used in edge AI scenarios such as smart cameras, robots, and drones.

[0003] However, Atlas 200AI falls under the category of hardware decoding. When the protocol frame does not fully conform to the decoding protocol, or when there are many erroneous frames, the decoding efficiency is low or decoding is impossible. Furthermore, for image information with parameter frames, the hardware decoding module cannot extract the parameter information. This invention uses the ffmpeg decoding library to obtain parameter frame information and data frame information from the original frame, obtains the complete image frame data packet, and transmits the decodeable complete data packet to the hardware decoding module to complete the decoding function. Simultaneously, to solve the problem of providing an interface for upper-layer applications to call, it is provided to the upper layer in the form of a dynamic library. Moreover, to prevent the hardware from failing to release memory resources due to decoding timeouts caused by erroneous frames, monitoring methods are used to release memory in a timely manner to avoid memory leaks. Summary of the Invention

[0004] (I) Purpose of the Invention

[0005] The purpose of this invention is to provide a video decoding method compatible with multiple bitstreams by adopting a hardware-software combined decoding method based on acceleration modules, thereby solving the video decoding problem under the Linux system.

[0006] (II) Technical Solution

[0007] To address the aforementioned technical problems, this invention provides a decoding method compatible with multiple bitstreams, comprising the following steps:

[0008] S1: Porting ffmpeg to the acceleration module platform;

[0009] S2: Set up the interface for initializing resource calls;

[0010] S3: Use the ffmpeg dynamic library to obtain a complete compressed image data packet containing parameter frames;

[0011] S4: Filter the complete bitstream;

[0012] S5: Perform hardware decoding on the filtered bitstream;

[0013] S6: Set an error callback function to check the decoding status;

[0014] S7: Return the decoded image to the upper-layer application through the interface function;

[0015] S8: Release the decoding channel resources.

[0016] Step S1 includes the following processes:

[0017] S11: Configure ffmpeg compilation properties, and configure parameters according to platform type, CPU type, codec properties, format conversion properties, and cross-compilation properties;

[0018] S12: Modify the acodec.h file to add the parameter frame length sei_len and the array sei_buf, and determine the parameter frame size SEI_BUF_SIZE according to the application layer requirements;

[0019] S13: Add a function to get the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: get the parameter size in the function, assign size to sei_len, check if size is less than or equal to SEI_BUF_SIZE, if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array index i of the copied sei_buf is the index of the gb array divided by 8, that is, sei_buf[i] = gb_buf[index / 8];

[0020] S14: Execute the configure command, which will generate the decoding libraries libavcodec, libavformat, libavutil, and libswscale in the lib subfolder of the configuration folder.

[0021] S15: Copy the dynamic library to the / usr / lib path on the decoding board.

[0022] In step S2, the calling interface is set as an external function, the parameters and return value are set according to the protocol, and the decoding resources are initialized. The process is as follows:

[0023] S21: Initialize the semaphore mutex;

[0024] S22: Create an initialization resource setup thread. The thread's main logic setup process is as follows:

[0025] S221: Set the hardware decoding device serial number;

[0026] S222: Use library functions to create a hardware decoding context;

[0027] S223: Use library functions to create a hardware-decoded data stream object;

[0028] S224: Create a decoding thread;

[0029] S225: Initialize ffmpeg resources;

[0030] S226: Create a network data receiving thread and put the received data into a circular buffer array;

[0031] S227: Initialize the accelerated decoding module resources, the process is as follows:

[0032] S2271: Create decoding channel attributes. If creation is successful, proceed to the next step; otherwise, exit with an error.

[0033] S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0034] S2273: Set the decoding thread number;

[0035] S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0036] S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0037] S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0038] S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0039] S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0040] S2279: Create a decoding channel based on the decoding channel attributes;

[0041] S228: Periodically check if the network has received the image array, i.e., if the circular buffer array has data. If it does, take the data according to the frame length specified in the protocol and proceed to the next process; otherwise, continue to query and wait for network data.

[0042] Step S3 includes the following processes:

[0043] S31: Initialize the decoding library usage environment

[0044] First, set the decoder type to HEVC / h265; then, initialize the decoder context and dynamically allocate image frame storage space.

[0045] S32: Create an ffmpeg thread to retrieve data packets, the process is as follows:

[0046] S321: Obtain the pointer to the first address of the image data array read from the network, bufPtr, and its length, bufLen;

[0047] S322: Determine if the current data length bufLen is greater than 0. If it is, continue to the next step; otherwise, wait for the network to send data next time.

[0048] S323: Transmit the pointer to the first address of the data array bufPtr and the length bufLen to the software decoding module. Use the library function av_parser_parse2 to segment the data into frames. If a complete image frame data packet can be successfully obtained from the array, record it and continue to the next step; otherwise, exit the current decoding process.

[0049] S324: Save the length ret of the data packet for the image frame segmentation in the current data array, remove the length ret of the image frame data packet for the current segmentation from the total length bufLen of the data array, and move the starting address pointer bufPtr forward by ret times;

[0050] S325: Place the complete image data packet from this segmentation into the queue to be decoded.

[0051] Step S4 includes the following processes:

[0052] S41: Set the encoding type of the bitstream obtained by the upper-layer application calling the interface;

[0053] S42: Determine if the encoding type is a bitstream without I-frames, then initiate mode 1 for filtering. The filtering process for mode 1 is as follows:

[0054] S421: Check if the type of the current channel's frame is consistent with the previously saved frame type. If yes, proceed to the next step; otherwise, jump to S4224.

[0055] S422: Set the identifier indertIDR for inserting I-frames to 1;

[0056] S423: Determine whether the type value of the current frame is 0xff, 0x00, 0xaa, or 0xf0;

[0057] S424: If yes, save the type of the current frame to type as the latest frame type flag; otherwise, exit the filtering process.

[0058] S425: Get the length of the current frame and save it to the inserted frame length idrsize;

[0059] S426: Traverse the data of the current frame, with frame position i starting from the first data and ending at idrsize-3 data;

[0060] S427: Determine if the data at position i in the current frame is 0x00, the second data i+1 is 0x00, and the third data i+2 is 0x01. If so, continue to the next step; otherwise, jump to S426 to continue traversing.

[0061] S428: Determine if the data at position i+3 of the current frame is 0x40, 0x42, or 0x44. If yes, continue to the next step; otherwise, jump to S4212.

[0062] S429: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4211.

[0063] S4210: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0064] S4211: Set the I-frame flag idrFalg to 1;

[0065] S4212: Determine if the data at position i+3 of the current frame is 0x4E. If it is, proceed to the next step; otherwise, jump to S4216.

[0066] S4213: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4215.

[0067] S4214: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0068] S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start;

[0069] S4216: Determine if the data at position i+3 of the current frame is 0x26. If it is, proceed to the next step; otherwise, jump to S4220.

[0070] S4217: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4219.

[0071] S4218: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0072] S4219: Increment the number of P-frames in the current frame by 1 (isliceNumber), and set the frame count to 0.

[0073] S4220: Modify the current frame type to 0x02, that is, set the data at position i+3 to 0x02;

[0074] S4221: Determine if the data at position i+3 of the current frame is 0x02. If it is, continue to determine if the parameter frame flag seiFlag is 0. If it is, set the insert I-frame flag insertIDR to 1, and end the filtering process while waiting for the next received message.

[0075] S4222: Determine if the representation of the inserted I-frame is 1. If it is, assemble the frame according to the saved frame type. While assembling the frame, it is necessary to determine the number of current P-frames and the number of isliceNumbers, and fill in the frame content for different numbers; otherwise, proceed to the next step.

[0076] S4223: Modify the frame content according to the saved frame type, and modify the position of the current frame according to the number of current P frames and the number of isliceNumber.

[0077] S4224: Exit the process if the current frame is not a parameter frame or there is no I-frame or no P-frame;

[0078] S4225: Dynamically allocate memory for the image frame. Based on the filtering and framing process described above, copy the parameter information of the frame and the I-frame information within the group as needed. If the frame needs to insert an I-frame, skip this process and proceed directly to the next step.

[0079] S4226: The filtered frame content will not be pushed into the allocated image frame memory array;

[0080] S43: Determine if the encoding type is a standard bitstream containing I-frames, then initiate mode 2 for filtering. The filtering process for mode 2 is as follows:

[0081] S431: Dynamically allocate memory space inbuffer for image frames;

[0082] S432: Copy the size of the current frame (size) into the size of the array to be decoded (inbuffersize);

[0083] S433: Use the library function altMemcpy to copy the data of the middle frame to inbuffer, with a copy size of size;

[0084] S44: Increase seiCount by 1 to record the number of parameter frames of the data structure to be decoded;

[0085] S45: Determine if the current size of seiCount is greater than the macro-defined maximum number of parameter frames MAX_SEI_NUM. If it is, clear the current number of parameter frames seiCount to zero; otherwise, proceed to the next step.

[0086] S46: Save the size of the current parameter frame, seiSize, into the data structure to be decoded, and at the same time copy the content of the parameter frame into the parameter array, seiData, of the data structure to be decoded.

[0087] In step S5, the complete data packet filtered by ffmpeg is hardware decoded. Hardware decoding includes the following process:

[0088] S51: Set the decoding callback function, the process is as follows:

[0089] S511: Use library functions to obtain the decoded normal frame output. If the result is 0, proceed to the next step; otherwise, exit with an error.

[0090] S512: Use library functions to obtain image attribute data output;

[0091] S513: Use library functions to retrieve image size, width, and height parameter data from the output;

[0092] S514: Copy the image data and parameter data to the user-returned array g_send_buf;

[0093] S515: Unlocks the semaphore mutex, allowing user access to the returned array g_send_buf;

[0094] S52: Obtain the complete image data packet from step S4;

[0095] S53: Send the data packet to the hardware decoding module for decoding. The decoding process is as follows:

[0096] S531: Use library functions to dynamically allocate image storage space;

[0097] S532: Create an input image stream attribute object;

[0098] S533: Set the input image stream property object;

[0099] S534: Dynamically allocates memory space for image output;

[0100] S535: Create an output image attribute object;

[0101] S536: Set the output image attributes based on the image's data, size, and format;

[0102] S537: Use library functions to send image frame data to the hardware decoding module by passing the decoding channel attributes, input image stream attributes, and output image attributes as parameters;

[0103] S538: Release the allocated image memory space;

[0104] S539: Destroy the input image stream object resource.

[0105] In step S6, during the decoding process, it is checked in real time whether the image frames can be decoded successfully. If not, resources need to be released. The process is as follows:

[0106] S61: Set the error callback function to callbackErr, with the parameter being a pointer to an exception type object;

[0107] S62: Use the library function to obtain the decoding task number taskid from the exception information;

[0108] S63: Obtain the image frame number (frameid) from the decoded stream in the abnormal situation information;

[0109] S64: Obtain the device number from the abnormal situation decoding device;

[0110] S65: Check if the device ID and task taskd in the previous initialization settings are consistent. If they are, use the image frame number frameid to allocate memory space in step S5.

[0111] In step S7, after obtaining the hardware-decoded image, the image data is returned through the interface function, as follows:

[0112] S71: Set up the readbuf function to provide an interface for acquiring image data, with the following content:

[0113] S711: Semaphore locking to prevent read / write conflicts on user-returned arrays;

[0114] S712: Copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf;

[0115] S713: Returns the length of the image data;

[0116] S72: Sets up a function void* getbufAddress to obtain the address of the image data array, which returns the address of g_read_buf.

[0117] The process of step S8 is as follows:

[0118] S81: Report the decoding thread ID and data stream ID described by calling library functions;

[0119] S82: Set the decoding thread flag to false;

[0120] S83: Release the decoding thread;

[0121] S84: Destroy the decode channel object;

[0122] S85: Set the decoding channel description pointer to null;

[0123] S86: Destroy the decoded stream description object;

[0124] S87: Destroy the decoded stream object;

[0125] S88: Sets the array object in the decoded stream to empty.

[0126] (III) Beneficial Effects

[0127] The aforementioned technical solution provides a multi-stream compatible video decoding method that can handle video decoding and transmission under Linux systems. This method has undergone algorithm verification and experimental testing. Results show that this solution effectively reduces decoding time by employing a software decoding method to assemble a complete data packet containing image frames and parameter frames, and then using the chip's hardware decoding module to complete the decoding process. Furthermore, a filtering mechanism is established to activate different decoding mechanisms for different bitstreams, thereby improving decoding efficiency. Detailed Implementation

[0128] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to examples.

[0129] The multi-stream compatible video decoding method in this embodiment includes the following steps:

[0130] S1: Porting ffmpeg to the acceleration module platform

[0131] S11: Configure ffmpeg compilation properties, including parameters based on platform type, CPU type, codec properties, format conversion properties, and cross-compilation properties.

[0132] S12: Modify the acodec.h file to add the parameter frame length sei_len and the array sei_buf, and determine the parameter frame size SEI_BUF_SIZE according to the application layer requirements;

[0133] S13: Add a function to get the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: get the parameter size in the function, assign size to sei_len, check if size is less than or equal to SEI_BUF_SIZE, if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array index i of the copied sei_buf is the index of the gb array divided by 8, that is, sei_buf[i] = gb_buf[index / 8];

[0134] S14: Execute the configure command, which will generate the decoding libraries libavcodec, libavformat, libavutil, and libswscale in the lib subfolder of the configuration folder.

[0135] S15: Copy the dynamic library to the / usr / lib path of the decoding board;

[0136] S2: Set up the resource call interface for initialization

[0137] Set the API call as an external function, configure the parameters and return value according to the protocol, and initialize the decoding resources. The specific steps are as follows:

[0138] S21: Initialize the semaphore mutex;

[0139] S22: Create an initialization resource setup thread. The main logic of the thread is as follows:

[0140] S221: Set the hardware decoding device serial number;

[0141] S222: Use library functions to create a hardware decoding context;

[0142] S223: Use library functions to create a hardware-decoded data stream object;

[0143] S224: Create a decoding thread;

[0144] S225: Initialize ffmpeg resources;

[0145] S226: Create a network data receiving thread and put the received data into a circular buffer array;

[0146] S227: Initialize the accelerated decoding module resources. The specific steps are as follows:

[0147] S2271: Create decoding channel attributes. If creation is successful, proceed to the next step; otherwise, exit with an error.

[0148] S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0149] S2273: Set the decoding thread number;

[0150] S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0151] S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0152] S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0153] S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0154] S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step; otherwise, exit with an error.

[0155] S2279: Create a decoding channel based on the decoding channel attributes;

[0156] S228: Periodically check whether the network has received the image array, i.e., whether the circular buffer array has data. If it does, take the data according to the frame length specified in the protocol and proceed to the next process; otherwise, continue to query and wait for network data.

[0157] S3: Use the ffmpeg dynamic library to obtain a complete compressed image data packet containing parameter frames.

[0158] S31: Initialize the decoding library usage environment

[0159] First, set the decoder type to HEVC (h265). Then, initialize the decoder context and dynamically allocate image frame storage space.

[0160] S32: Create an ffmpeg thread to acquire data packets. The specific thread flow is as follows:

[0161] S321: Obtain the pointer to the first address of the image data array read from the network, bufPtr, and its length, bufLen;

[0162] S322: Determine if the current data length bufLen is greater than 0. If it is, continue to the next step; otherwise, wait for the network to send data next time.

[0163] S323: Transmit the pointer to the first address of the data array bufPtr and the length bufLen to the software decoding module. Use the library function av_parser_parse2 to segment the data into frames. If a complete image frame data packet can be successfully obtained from the array, record it and continue to the next step; otherwise, exit the current decoding process.

[0164] S324: Save the length ret of the data packet of the image frame segmentation in the data array this time, remove the length ret of the data packet of the image frame segmentation this time from the total length bufLen of the data array, and move the first address pointer bufPtr forward by ret times.

[0165] S325: Place the complete image data packet from this segmentation into the queue to be decoded.

[0166] S4: Filter the complete bitstream

[0167] The specific steps are as follows:

[0168] S41: Set the encoding type of the bitstream obtained by the upper-layer application calling the interface;

[0169] S42: Determine if the encoding type is a bitstream without I-frames, then initiate mode 1 for filtering. The specific steps are as follows:

[0170] S421: Check if the type of the current channel's frame is consistent with the previously saved frame type. If yes, proceed to the next step; otherwise, jump to S4224.

[0171] S422: Set the identifier indertIDR for inserting I-frames to 1;

[0172] S423: Determine whether the type value of the current frame is 0xff, 0x00, 0xaa, or 0xf0;

[0173] S424: If yes, save the type of the current frame to type as the latest frame type flag; otherwise, exit the filtering process.

[0174] S425: Get the length of the current frame and save it to the inserted frame length idrsize;

[0175] S426: Traverse the data of the current frame, with frame position i starting from the first data and ending at idrsize-3 data;

[0176] S427: Determine if the data at position i in the current frame is 0x00, and the second data i+1 is 0x00, and the third data i+2 is 0x01. If so, continue to the next step; otherwise, jump to S426 to continue traversing.

[0177] S428: Determine if the data at position i+3 of the current frame is 0x40, 0x42, or 0x44. If yes, continue to the next step; otherwise, jump to S4212.

[0178] S429: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4211.

[0179] S4210: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0180] S4211: Set the I-frame flag idrFalg to 1;

[0181] S4212: Determine if the data at position i+3 of the current frame is 0x4E. If it is, proceed to the next step; otherwise, jump to S4216.

[0182] S4213: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4215.

[0183] S4214: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0184] S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start;

[0185] S4216: Determine if the data at position i+3 of the current frame is 0x26. If it is, proceed to the next step; otherwise, jump to S4220.

[0186] S4217: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4219.

[0187] S4218: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process;

[0188] S4219: Increment the number of P-frames in the current frame by 1 (isliceNumber), and set the frame count to 0.

[0189] S4220: Modify the current frame type to 0x02, that is, set the data at position i+3 to 0x02;

[0190] S4221: Determine if the data at position i+3 of the current frame is 0x02. If it is, continue to determine if the parameter frame flag seiFlag is 0. If it is, set the insert I-frame flag insertIDR to 1, and end the filtering process while waiting for the next received message.

[0191] S4222: Determine if the representation of the inserted I-frame is 1. If it is, assemble the frame according to the saved frame type. While assembling the frame, it is necessary to determine the number of current P-frames and the number of isliceNumbers, and fill in the frame content for different numbers; otherwise, proceed to the next step.

[0192] S4223: Modify the frame content according to the saved frame type, and modify the position of the current frame according to the number of current P frames and the number of isliceNumber.

[0193] S4224: Exit the process if the current frame is not a parameter frame or there is no I-frame or no P-frame;

[0194] S4225: Dynamically allocate memory for the image frame. Based on the filtering and framing process described above, copy the parameter information of the frame and the I-frame information within the group as needed. If the frame needs to insert an I-frame, skip this process and proceed directly to the next step.

[0195] S4226: The filtered frame content will not be pushed into the allocated image frame memory array;

[0196] S43: Determine if the encoding type is a standard bitstream containing I-frames, then initiate mode 2 for filtering. The specific process is as follows:

[0197] S431: Dynamically allocate memory space inbuffer for image frames;

[0198] S432: Copy the size of the current frame (size) into the size of the array to be decoded (inbuffersize);

[0199] S433: Use the library function altMemcpy to copy the data of the middle frame to inbuffer, with a copy size of size;

[0200] S44: Increase seiCount by 1 to record the number of parameter frames of the data structure to be decoded;

[0201] S45: Determine if the current size of seiCount is greater than the macro-defined maximum number of parameter frames MAX_SEI_NUM. If it is, clear the current number of parameter frames seiCount to zero; otherwise, proceed to the next step.

[0202] S46: Save the size of the current parameter frame, seiSize, into the data structure to be decoded, and at the same time copy the content of the parameter frame into the parameter array seiData of the data structure to be decoded.

[0203] S5: Hardware decoding of the filtered bitstream

[0204] Perform hardware decoding on the complete data packet filtered by ffmpeg. The specific steps are as follows:

[0205] S51: Set the decoding callback function, as follows:

[0206] S511: Use library functions to obtain the decoded normal frame output. If the result is 0, proceed to the next step; otherwise, exit with an error.

[0207] S512: Use library functions to obtain image attribute data output;

[0208] S513: Use library functions to retrieve image size, width, and height parameter data from the output;

[0209] S514: Copy the image data and parameter data to the user-returned array g_send_buf;

[0210] S515: Unlocks the semaphore mutex, allowing user access to the returned array g_send_buf;

[0211] S52: Obtain the complete image data packet from step S4;

[0212] S53: Send the data packet to the hardware decoding module for decoding, as follows:

[0213] S531: Use library functions to dynamically allocate image storage space;

[0214] S532: Create an input image stream attribute object;

[0215] S533: Set the input image stream property object;

[0216] S534: Dynamically allocates memory space for image output;

[0217] S535: Create an output image attribute object;

[0218] S536: Set the output image attributes based on the image's data, size, and format;

[0219] S537: Use library functions to send image frame data to the hardware decoding module by passing the decoding channel attributes, input image stream attributes, and output image attributes as parameters;

[0220] S538: Release the allocated image memory space;

[0221] S539: Destroy the input image stream object resource;

[0222] S6: Set an error callback function to check the decoding status

[0223] During the decoding process, monitor in real time whether the image frames can be decoded successfully. If not, resources need to be released. The specific steps are as follows:

[0224] S61: Set the error callback function to callbackErr, with the parameter being a pointer to an exception type object;

[0225] S62: Use the library function to obtain the decoding task number taskid from the exception information;

[0226] S63: Obtain the image frame number (frameid) from the decoded stream in the abnormal situation information;

[0227] S64: Obtain the device number from the abnormal situation decoding device;

[0228] S65: Check if the device ID and task taskd in the previous initialization settings are consistent. If they are, use the image frame number frameid to allocate memory space in step S5.

[0229] S7: Return the decoded image to the upper-layer application via an interface function.

[0230] After obtaining the hardware-decoded image, the image data is returned through an interface function. The specific steps are as follows:

[0231] S71: Set up the readbuf function to provide an interface for acquiring image data. The details are as follows:

[0232] S711: Semaphore locking to prevent read / write conflicts on user-returned arrays;

[0233] S712: Copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf;

[0234] S713: Returns the length of the image data;

[0235] S72: Provides a function `void* getbufAddress` to retrieve the address of the image data array, which returns the address of `g_read_buf`.

[0236] S8: Release decoding channel resources

[0237] The specific steps are as follows:

[0238] S81: Report the decoding thread ID and data stream ID described by calling library functions;

[0239] S82: Set the decoding thread flag to false (do not decode);

[0240] S83: Release the decoding thread;

[0241] S84: Destroy the decode channel object;

[0242] S85: Set the decoding channel description pointer to null;

[0243] S86: Destroy the decoded stream description object;

[0244] S87: Destroy the decoded stream object;

[0245] S88: Sets the array object in the decoded stream to empty.

[0246] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A decoding method compatible with multiple bitstreams, characterized in that, Includes the following steps: S1: Porting ffmpeg to the acceleration module platform; S2: Set up the interface for initializing resource calls; S3: Use the ffmpeg dynamic library to obtain a complete compressed image data packet containing parameter frames; S4: Filter the complete bitstream; S5: Perform hardware decoding on the filtered bitstream; S6: Set an error callback function to check the decoding status; S7: Return the decoded image to the upper-layer application through the interface function; S8: Release the decoding channel resources; Step S1 includes the following processes: S11: Configure ffmpeg compilation properties, and configure parameters according to platform type, CPU type, codec properties, format conversion properties, and cross-compilation properties; S12: Modify the acodec.h file to add the parameter frame length sei_len and the array sei_buf, and determine the parameter frame size SEI_BUF_SIZE according to the application layer requirements; S13: Add a function to get the parameter frame in the decode_nal_sei_prefix function in the hevc_sei.c file: get the parameter size in the function, assign size to sei_len, check whether size is less than or equal to SEI_BUF_SIZE, if the condition is met, copy the data in the context parameter array gb to sei_buf, the length of the copy is size, and the array index i of the copied sei_buf is the index of the gb array divided by 8, that is, sei_buf[i]=gb_buf[index / 8]; S14: Execute the configure command, which will generate the decoding libraries libavcodec, libavformat, libavutil, and libswscale in the lib subfolder of the configuration folder; S15: Copy the dynamic library to the / usr / lib path of the decoding board; In step S2, the calling interface is set as an external function, the parameters and return value are set according to the protocol, and the decoding resources are initialized. The process is as follows: S21: Initialize the semaphore mutex; S22: Create an initialization resource setup thread. The thread's main logic setup process is as follows: S221: Set the hardware decoding device serial number; S222: Use library functions to create a hardware decoding context; S223: Use library functions to create a hardware-decoded data stream object; S224: Create a decoding thread; S225: Initialize ffmpeg resources; S226: Create a network data receiving thread and put the received data into a circular buffer array; S227: Initialize the accelerated decoding module resources, the process is as follows: S2271: Create decoding channel attributes. If creation is successful, proceed to the next step; otherwise, exit with an error. S2272: Set the decoding channel ID number. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2273: Set the decoding thread number; S2274: Set the decoding channel callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2275: Set the decoding channel error exception callback function. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2276: Set the decoding type of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2277: Set the output image format of the decoding channel. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2278: Set the decoding channel output mode. If the setting is successful, proceed to the next step; otherwise, exit with an error. S2279: Create a decoding channel based on the decoding channel attributes; S228: Periodically check whether the network has received the image array, i.e., whether the circular buffer array has data. If it does, take the data according to the frame length specified in the protocol and proceed to the next process; otherwise, continue to query and wait for network data. Step S3 includes the following processes: S31: Initialize the decoding library usage environment First, set the decoder type to HEVC / h265; then, initialize the decoder context and dynamically allocate image frame storage space. S32: Create an ffmpeg thread to retrieve data packets, the process is as follows: S321: Obtain the pointer to the first address of the image data array read from the network, bufPtr, and its length, bufLen; S322: Determine if the current data length bufLen is greater than 0. If it is, continue to the next step; otherwise, wait for the network to send data next time. S323: Transmit the pointer to the first address of the data array bufPtr and the length bufLen to the software decoding module. Use the library function av_parser_parse2 to segment the data into frames. If a complete image frame data packet can be successfully obtained from the array, record it and continue to the next step; otherwise, exit the current decoding process. S324: Save the length ret of the data packet of the image frame segmentation in the data array this time, remove the length ret of the data packet of the image frame segmentation this time from the total length bufLen of the data array, and move the first address pointer bufPtr forward by ret times; S325: Place the complete image data packet from this segmentation into the decoding queue; Step S4 includes the following processes: S41: Set the encoding type of the bitstream obtained by the upper-layer application calling the interface; S42: Determine if the encoding type is a bitstream without I-frames, then initiate mode 1 for filtering. The filtering process for mode 1 is as follows: S421: Check if the type of the current channel's frame is consistent with the previously saved frame type. If yes, proceed to the next step; otherwise, jump to S4224. S422: Set the identifier indertIDR for inserting I-frames to 1; S423: Determine whether the type value of the current frame is 0xff, 0x00, 0xaa, or 0xf0; S424: If yes, save the type of the current frame to type as the latest frame type flag; otherwise, exit the filtering process. S425: Get the length of the current frame and save it to the inserted frame length idrsize; S426: Traverse the data of the current frame, with frame position i starting from the first data and ending at idrsize-3 data; S427: Determine if the data at position i in the current frame is 0x00, the second data i+1 is 0x00, and the third data i+2 is 0x01. If so, continue to the next step; otherwise, jump to S426 to continue traversing. S428: Determine if the data at position i+3 of the current frame is 0x40, 0x42, or 0x44. If yes, continue to the next step; otherwise, jump to S4212. S429: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4211. S4210: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process; S4211: Set the I-frame flag idrFalg to 1; S4212: Determine if the data at position i+3 of the current frame is 0x4E. If it is, proceed to the next step; otherwise, jump to S4216. S4213: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4215. S4214: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process; S4215: Set seiFlag to 1, record the current frame position i and save it to the frame start position u32start, and record the size of the data frame as the current frame size minus u32start; S4216: Determine if the data at position i+3 of the current frame is 0x26. If it is, proceed to the next step; otherwise, jump to S4220. S4217: Determine if the parameter flag seiFlag is 1. If it is, proceed to the next step; otherwise, jump to S4219. S4218: Set the insertIDR flag for inserting I-frames to 1 and exit the filtering process; S4219: Increment the number of P-frames in the current frame by 1 (isliceNumber), and set the frame count to 0. S4220: Modify the current frame type to 0x02, that is, set the data at position i+3 to 0x02; S4221: Determine if the data at position i+3 of the current frame is 0x02. If it is, continue to determine if the parameter frame flag seiFlag is 0. If it is, set the insert I-frame flag insertIDR to 1, and end the filtering process while waiting for the next received message. S4222: Determine if the representation of the inserted I-frame is 1. If it is, assemble the frame according to the saved frame type. At the same time, it is necessary to determine the number of current P-frames and the number of isliceNumber, and fill in the frame content for different numbers. Otherwise, proceed to the next step; S4223: Modify the frame content according to the saved frame type, and modify the position of the current frame according to the number of current P frames and the number of isliceNumber. S4224: Exit the process if the current frame is not a parameter frame or there is no I-frame or no P-frame; S4225: Dynamically allocate memory for the image frame. Based on the filtering and framing process described above, copy the parameter information of the frame and the I-frame information within the group as needed. If the frame needs to insert an I-frame, skip this process and proceed directly to the next step. S4226: The filtered frame content will not be pushed into the allocated image frame memory array; S43: Determine if the encoding type is a standard bitstream containing I-frames, then initiate mode 2 for filtering. The filtering process for mode 2 is as follows: S431: Dynamically allocate memory space inbuffer for image frames; S432: Copy the size of the current frame (size) into the size of the array to be decoded (inbuffersize); S433: Use the library function altMemcpy to copy the data of the middle frame to inbuffer, with a copy size of size; S44: Increase seiCount by 1 to record the number of parameter frames of the data structure to be decoded; S45: Determine if the current size of seiCount is greater than the macro-defined maximum number of parameter frames MAX_SEI_NUM. If it is, clear the current number of parameter frames seiCount to zero; otherwise, proceed to the next step. S46: Save the size of the current parameter frame, seiSize, into the data structure to be decoded, and at the same time copy the content of the parameter frame into the parameter array seiData of the data structure to be decoded; In step S5, the complete data packet filtered by ffmpeg is hardware decoded. Hardware decoding includes the following process: S51: Set the decoding callback function, the process is as follows: S511: Use library functions to obtain the decoded normal frame output. If the result is 0, proceed to the next step; otherwise, exit with an error. S512: Use library functions to obtain image attribute data output; S513: Use library functions to retrieve image size, width, and height parameter data from the output; S514: Copy the image data and parameter data to the user-returned array g_send_buf; S515: Unlocks the semaphore mutex, allowing user access to the returned array g_send_buf; S52: Obtain the complete image data packet from step S4; S53: Send the data packet to the hardware decoding module for decoding. The decoding process is as follows: S531: Use library functions to dynamically allocate image storage space; S532: Create an input image stream attribute object; S533: Set the input image stream property object; S534: Dynamically allocates memory space for image output; S535: Create an output image attribute object; S536: Set the output image attributes based on the image's data, size, and format; S537: Use library functions to send image frame data to the hardware decoding module by passing the decoding channel attributes, input image stream attributes, and output image attributes as parameters; S538: Release the allocated image memory space; S539: Destroy the input image stream object resource; In step S6, during the decoding process, it is checked in real time whether the image frames can be decoded successfully. If not, resources need to be released. The process is as follows: S61: Set the error callback function to callbackErr, with the parameter being a pointer to an exception type object; S62: Use the library function to obtain the decoding task number taskid from the exception information; S63: Obtain the image frame number (frameid) from the decoded stream in the abnormal situation information; S64: Obtain the device number from the abnormal situation decoding device; S65: Check if the device ID and task taskd in the previous initialization settings are consistent. If they are, use the image frame number frameid to allocate memory space in step S5.

2. The multi-stream compatible decoding method as described in claim 1, characterized in that, In step S7, after obtaining the hardware-decoded image, the image data is returned through the interface function, as follows: S71: Set up the readbuf function to provide an interface for acquiring image data, with the following content: S711: Semaphore locking to prevent read / write conflicts on user-returned arrays; S712: Copy the image data array g_send_buf decoded in step S5 to the temporary data array g_read_buf; S713: Returns the length of the image data; S72: Sets up a function void* getbufAddress to obtain the address of the image data array, which returns the address of g_read_buf.

3. The multi-stream compatible decoding method as described in claim 2, characterized in that, The process of step S8 is as follows: S81: Report the decoding thread ID and data stream ID described by calling library functions; S82: Set the decoding thread flag to false; S83: Release the decoding thread; S84: Destroy the decode channel object; S85: Set the decoding channel description pointer to null; S86: Destroy the decoded stream description object; S87: Destroy the decoded stream object; S88: Sets the array object in the decoded stream to empty.