A method for real-time playing of video decoding under QT environment

Abstract

The application discloses a video decoding real-time playing method in QT environment, which comprises the following steps: S1, creating a compiling environment; S2, initializing ffmpeg stream pulling and decoding resources; S3, creating a stream pulling main thread; S4, decoding a data packet object obtained by the stream pulling; S5, carrying out format conversion on the decoded image frame; S6, obtaining the image frame after the format conversion and displaying in real time; and S7, releasing the decoding channel resources. The application adopts a soft decoding method, effectively reduces the decoding time, simultaneously adopts an RTSP video transmission protocol to obtain a video stream, effectively reduces playing delay, and guarantees the fluency of video playing.

Description

Technical Field

[0001] This invention belongs to the field of video decoding technology under cross-platform systems. In application scenarios where the system is uncertain, this invention designs and implements a real-time video decoding and playback method compatible with multiple system types in a QT environment, taking into account the decoding characteristics of FFmpeg and combining it with the QT module to design an image display scheme. The invention first uses the FFmpeg software decoding library to pull the network bitstream to RTSP, effectively acquiring the video stream. Then, it decodes the bitstream according to its encoding characteristics and displays the decoded image, thereby completing the decoding, playback, and display process. Background Technology

[0002] Qt is a cross-platform graphical user interface application development framework for building high-performance desktop, mobile, and web applications. It can also be used to develop non-GUI programs, such as console tools and servers. Qt is an object-oriented framework that, using special code generation extensions and macros, is easily extensible and allows for true component programming.

[0003] Ffmpeg is an open-source computer program used to record, convert, and stream digital audio and video. It provides a complete solution for recording, converting, and streaming audio and video. It includes highly advanced audio / video codec libraries.

[0004] Real-Time Streaming Protocol (RTSP) is an application layer protocol in the TCP / IP protocol suite. This protocol defines how one-to-many applications can efficiently transmit multimedia data over an IP network.

[0005] In projects with unclear requirements, choosing a cross-platform tool for development can save development costs. Combining Qt and ffmpeg can solve multi-platform compatibility issues, facilitate easier module porting, and utilize the RTSP video transmission protocol to address real-time video playback across platforms. Summary of the Invention

[0006] (I) Purpose of the Invention

[0007] The purpose of this invention is to solve the problem of real-time video decoding and playback under multiple systems, and to provide a real-time video decoding and playback method in a QT environment, which adopts a decoding method based on a combination of QT and ffmpeg.

[0008] (II) Technical Solution

[0009] To address the aforementioned technical problems, this invention provides a real-time video decoding and playback method in a QT environment, comprising the following steps:

[0010] S1: Create the compilation environment;

[0011] S2: Initialize ffmpeg streaming and decoding resources;

[0012] S3: Create the main thread for pulling the stream;

[0013] S4: Decode the data packet object obtained from the pull stream;

[0014] S5: Convert the format of the decoded image frames;

[0015] S6: Acquire the converted image frames and display them in real time;

[0016] S7: Release the decoding channel resources.

[0017] The process of creating the compilation environment in step S1 is as follows:

[0018] S11: Obtain the dynamic library files libacodec.lib, libavformat.lib, libavutil.lib, and libswscale.lib from ffmpeg and copy them to the working directory lib.

[0019] S12: Copy the ffmpeg include header file to the include directory of the project;

[0020] S13: Add the environment variable FFMPEGLIBPATH to the system and set this environment variable to the lib directory under the project directory;

[0021] S14: Add an external library to the Qt project file. Add the library file from step S11, using the path indicated in step S13.

[0022] In step S2, a new thread class named decode is created in the qt project directory. This class inherits from QThread.

[0023] In step S2, the process of implementing the streaming initialization resource InitVideo in the thread class decode is as follows:

[0024] S201: Use the library function avformat_network_init to initialize the network;

[0025] S202: Use the library function avformat_alloc_context to create a streaming context object decoder_avfc;

[0026] S203: Check if the context object decoder_avfc is empty. If it is empty, exit the process; otherwise, proceed to the next step.

[0027] S204: Use the library function av_log_set_level to enable logging functionality;

[0028] S205: Use the library function av_dict_set to set the avdic parameter of rstp;

[0029] S206: Use the library function avformat_open_input to open the preset RTSP streaming URL;

[0030] S207: Determine whether the result of step S206 is successful. If it is unsuccessful, record the error message and exit the process; otherwise, proceed to the next step.

[0031] S208: Check if the parameter avdic is empty. If not, use the library function av_dict_free to release the object's resources; otherwise, proceed to the next step.

[0032] S209: Use the library function avformat_find_stream_info to obtain the video stream. If the acquisition fails, use the library function avformat_close_input to close the context object decoder_avfc and release resources. Otherwise, proceed to the next step.

[0033] In step S2, the process of decoding and initializing resources in the thread class decode is as follows:

[0034] S211: Use the library function av_packet_alloc to create a packet object pkt. If creation is successful, proceed to the next step; otherwise, exit the process.

[0035] S212: Use the library function avcodec_find_decoder to create a decoder codec. If creation is successful, proceed to the next step; otherwise, exit the process.

[0036] S213: Use the library function av_parser_init to create a packet analysis object parser. If creation is successful, proceed to the next step; otherwise, exit the process.

[0037] S214: Use the library function avcodec_alloc_context3 to create a decoding environment context object. If creation is successful, proceed to the next step; otherwise, exit the process.

[0038] S215: Use the library function avcodec_oepn2 to open the decoder. If it fails, exit the process; otherwise, proceed to the next step.

[0039] S216: Use the library function av_frame_alloc to create an image frame object frame. If it fails, exit the process; otherwise, proceed to the next step.

[0040] In step S3, the process of creating the main pull stream thread is as follows:

[0041] S31: Perform main loop for current pulling;

[0042] S32: Check if the parameter setting flag is successful. If successful, proceed to the next step; otherwise, delay for 10µs.

[0043] S33: Determine whether the streaming initialization step InitVideo in process S2 was successful. If it was, exit the connection establishment loop and continue to the next step; otherwise, continue the connection operation.

[0044] S34: Enter the video stream acquisition loop;

[0045] S35: Create a streaming data packet;

[0046] S36: Initialize packet using the library function av_init_packet;

[0047] S37: Set the packet's data pointer to a null pointer;

[0048] S38: Set the packet's data size to 0;

[0049] S39: Check if the streaming context decode_avfc is empty. If it is, return failure and exit the process; otherwise, proceed to the next step.

[0050] S310: Use the library function av_read_frame to retrieve the streaming video data packets opened by decode_avfc. If opening fails, release the packet object; otherwise, store the data in the packet and proceed to the next step.

[0051] S311: Determine if the stream index of the packet object is the macro-defined video stream vdieo_Stream. If it is, proceed to the next decoding process and release the packet object; otherwise, release the packet object directly.

[0052] In step S4, the process of decoding the data packet object obtained by the pull stream is as follows:

[0053] S41: Use the library function avcode_send_packet to send the video stream data packet object (packet) obtained in process S3 to the decoder. If the sending fails, report an error and exit the process; otherwise, proceed to the next step.

[0054] S42: Enter the loop to acquire decoded image frames;

[0055] S43: Determine whether the result ret of the acquired image frame is greater than or equal to zero. If it is, proceed to the next step; otherwise, exit the loop process of acquiring image frames.

[0056] S44: Use the library function avcodec_receive_frame to get the video frame where the context object dec_ctx is located. If the acquisition is successful, it is stored in the image frame object frame and the next step is performed. If it fails, an error message is printed and the process is exited.

[0057] S45: Use the system function fflush to force the decoder to output image frames.

[0058] In step S5, the process of format conversion of the decoded image frame is as follows:

[0059] S51: Use the library function av_frame_alloc to create an RGB image frame rgbFrame;

[0060] S52: Use the library function avpicture_get_size to calculate the data size numBytes required to convert to rgb24 format;

[0061] S53: Use the library function av_malloc to dynamically allocate an array rgbBuffer to store RGB image data, with a size of numBytes*sizeof(uint8_t);

[0062] S54: Use the library function avpicture_fill to index the array of RGB image frames rgbFrame into rgbBuffer;

[0063] S55: Check if the conversion context object image_convert_ctx is empty. If not, use the library function sws_freeContext to release the object; otherwise, proceed to the next step.

[0064] S56: Use the library function sws_getContext to create a conversion context pair image_convert_ctx;

[0065] S57: Determine if the conversion context pair image_convert_ctx is empty. If it is, exit the process; otherwise, proceed to the next step.

[0066] S58: Use the library function sws_scale to convert the image frame and store the data in rgbFrame. Store the conversion result in result.

[0067] S59: Determine if the conversion result is successful. If it is, proceed to the next step; otherwise, exit the process.

[0068] S510: Send the converted image frame rgbBuffer to the interface.

[0069] In step S6, the process of acquiring the converted image frame and displaying it in real time is as follows:

[0070] S61: Determine if the image frame object pointer is null. If it is, exit the process; otherwise, proceed to the next step.

[0071] S62: Create an image display object, which is of type QImage;

[0072] S63: Create a pixel map object pix, which is of type QPixmap;

[0073] S64: Get the pointer to the display image object of the QLabel class, image_display;

[0074] S65: Use the system function setPixmap to set the image object pointer image_display to display the image. The parameter is the return object of the pix from the image function.

[0075] In step S7, the process of releasing the decoding channel resources is as follows:

[0076] S71: Use the library function av_parser_close to close the packet analysis object parser;

[0077] S72: Use the library function avcodec_free_context to release the decoding context object;

[0078] S73: Use the library function avformat_close_input to release the streaming context object decoder_avfc;

[0079] S74: Use the library function av_packet_free to release the packet object;

[0080] S75: Use the library function av_frame_free to release the image frame object;

[0081] S76: Use the library function av_free to release the converted RGB array rgbBuffer.

[0082] (III) Beneficial Effects

[0083] The above-described technical solution provides a real-time video decoding and playback method in a QT environment, which can solve video decoding and transmission problems across multiple systems. This method has undergone algorithm verification and experimental testing. Results show that this solution effectively reduces decoding time by employing a software decoding method to address image decoding issues, and by using the RTSP video transmission protocol to obtain the video stream, it effectively reduces playback latency and ensures smooth video playback. Detailed Implementation

[0084] To make the objectives, contents, and advantages of the present invention clearer, the specific embodiments of the present invention will be described in further detail below with reference to examples.

[0085] The real-time video decoding and playback method in the QT environment of this embodiment includes the following steps:

[0086] S1: Creating the compilation environment

[0087] S11: First, obtain the dynamic library files libacodec.lib, libavformat.lib, libavutil.lib, and libswscale.lib from ffmpeg (the file extensions of dynamic library files vary depending on the system, so adjust accordingly) and copy them to the working directory lib.

[0088] S12: Copy the ffmpeg include header file to the include directory of the project;

[0089] S13: Add the environment variable FFMPEGLIBPATH to the system and set this environment variable to the lib directory under the project directory;

[0090] S14: Add an external library to the Qt project file. Add the library file from step 1, using the path indicated in step 3.

[0091] S2: Initialize ffmpeg streaming and decoding resources

[0092] Create a new thread class `decode` in the Qt project directory. This class inherits from `QThread`. The specific steps for implementing the streaming initialization resource `InitVideo` in this class are as follows:

[0093] S201: Use the library function avformat_network_init to initialize the network;

[0094] S202: Use the library function avformat_alloc_context to create a streaming context object decoder_avfc;

[0095] S203: Check if the context object decoder_avfc is empty. If it is empty, exit the process; otherwise, proceed to the next step.

[0096] S204: Use the library function av_log_set_level to enable logging functionality;

[0097] S205: Use the library function av_dict_set to set the avdic parameter of rstp;

[0098] S206: Use the library function avformat_open_input to open the preset RTSP streaming URL;

[0099] S207: Determine whether the result of step 6 is successful. If it is unsuccessful, record the error message and exit the process; otherwise, proceed to the next step.

[0100] S208: Check if the parameter avdic is empty. If not, use the library function av_dict_free to release the object's resources; otherwise, proceed to the next step.

[0101] S209: Use the library function avformat_find_stream_info to obtain the video stream. If the acquisition fails, use the library function avformat_close_input to close the context object decoder_avfc and release resources. Otherwise, proceed to the next step.

[0102] The specific steps for implementing decoding and initializing resources in this class are as follows:

[0103] S211: Use the library function av_packet_alloc to create a packet object pkt. If creation is successful, proceed to the next step; otherwise, exit the process.

[0104] S212: Use the library function avcodec_find_decoder to create a decoder codec. If creation is successful, proceed to the next step; otherwise, exit the process.

[0105] S213: Use the library function av_parser_init to create a packet analysis object parser. If creation is successful, proceed to the next step; otherwise, exit the process.

[0106] S214: Use the library function avcodec_alloc_context3 to create a decoding environment context object. If creation is successful, proceed to the next step; otherwise, exit the process.

[0107] S215: Use the library function avcodec_oepn2 to open the decoder. If it fails, exit the process; otherwise, proceed to the next step.

[0108] S216: Use the library function av_frame_alloc to create an image frame object frame. If it fails, exit the process; otherwise, proceed to the next step.

[0109] S3: Create the main thread for pulling the stream

[0110] S31: Perform main loop for current pulling;

[0111] S32: Check if the parameter setting flag is successful. If successful, proceed to the next step; otherwise, delay for 10µs.

[0112] S33: Determine whether the InitVideo stream initialization step in process 2 was successful. If it was, exit the connection establishment loop and continue to the next step; otherwise, continue the connection operation.

[0113] S34: Enter the video stream acquisition loop;

[0114] S35: Create a streaming data packet;

[0115] S36: Initialize packet using the library function av_init_packet;

[0116] S37: Set the packet's data pointer to a null pointer;

[0117] S38: Set the packet's data size to 0;

[0118] S39: Check if the streaming context decode_avfc is empty. If it is, return failure and exit the process; otherwise, proceed to the next step.

[0119] S310: Use the library function av_read_frame to retrieve the streaming video data packets opened by decode_avfc. If opening fails, release the packet object; otherwise, store the data in the packet and proceed to the next step.

[0120] S311: Determine if the stream index of the packet object is the macro-defined video stream vdieo_Stream. If it is, proceed to the next decoding process and release the packet object; otherwise, release the packet object directly.

[0121] S4: Decode the data packet object obtained from the pull stream.

[0122] The specific steps are as follows:

[0123] S41: Use the library function avcode_send_packet to send the video stream data packet object (packet) obtained in process 3 to the decoder. If the sending fails, report an error and exit the process; otherwise, proceed to the next step.

[0124] S42: Enter the loop to acquire decoded image frames;

[0125] S43: Determine whether the result ret of the acquired image frame is greater than or equal to zero. If it is, proceed to the next step; otherwise, exit the loop process of acquiring image frames.

[0126] S44: Use the library function avcodec_receive_frame to get the video frame where the context object dec_ctx is located. If the acquisition is successful, it is stored in the image frame object frame and the next step is performed. If it fails, an error message is printed and the process is exited.

[0127] S45: Use the system function fflush to force the decoder to output image frames.

[0128] S5: Convert the format of the decoded image frames. The decoded image is in YUV format, which cannot be directly displayed in the QT interface. It needs to be converted from YUV format to RGB format. The specific steps are as follows:

[0129] S51: Use the library function av_frame_alloc to create an RGB image frame rgbFrame;

[0130] S52: Use the library function avpicture_get_size to calculate the data size numBytes required to convert to rgb24 format;

[0131] S53: Use the library function av_malloc to dynamically allocate an array rgbBuffer to store RGB image data. The size is numBytes*sizeof(uint8_t) (that is, one byte multiplied by the required data size, converted to bits).

[0132] S54: Use the library function avpicture_fill to index the array of RGB image frames rgbFrame into rgbBuffer;

[0133] S55: Check if the conversion context object image_convert_ctx is empty. If not, use the library function sws_freeContext to release the object; otherwise, proceed to the next step.

[0134] S56: Use the library function sws_getContext to create a conversion context pair image_convert_ctx;

[0135] S57: Determine if the conversion context pair image_convert_ctx is empty. If it is, exit the process; otherwise, proceed to the next step.

[0136] S58: Use the library function sws_scale to convert the image frame and store the data in rgbFrame. Store the conversion result in result.

[0137] S59: Determine if the conversion result is successful. If it is, proceed to the next step; otherwise, exit the process.

[0138] S510: Send the converted image frame rgbBuffer to the interface.

[0139] S6: Acquire the converted image frames and display them in real time.

[0140] The specific steps are as follows:

[0141] S61: Determine if the image frame object pointer is null. If it is, exit the process; otherwise, proceed to the next step.

[0142] S62: Create an image display object, which is of type QImage;

[0143] S63: Create a pixel map object pix, which is of type QPixmap;

[0144] S64: Get the pointer to the display image object of the QLabel class, image_display;

[0145] S65: Use the system function setPixmap to set the image object pointer image_display to display the image. The parameter is the return object of the pix from the image function.

[0146] S7: Release decoding channel resources

[0147] The specific steps are as follows:

[0148] S71: Use the library function av_parser_close to close the packet analysis object parser;

[0149] S72: Use the library function avcodec_free_context to release the decoding context object;

[0150] S73: Use the library function avformat_close_input to release the streaming context object decoder_avfc;

[0151] S74: Use the library function av_packet_free to release the packet object;

[0152] S75: Use the library function av_frame_free to release the image frame object;

[0153] S76: Use the library function av_free to release the converted RGB array rgbBuffer.

[0154] The above description is only a preferred embodiment of the present invention. It should be noted that for those skilled in the art, several improvements and modifications can be made without departing from the technical principles of the present invention, and these improvements and modifications should also be considered within the scope of protection of the present invention.

Claims

1. A method for real-time video decoding and playback in a QT environment, characterized in that, Includes the following steps: S1: Create the compilation environment; S2: Initialize ffmpeg streaming and decoding resources; S3: Create the main thread for pulling the stream; S4: Decode the data packet object obtained from the pull stream; S5: Convert the format of the decoded image frames; S6: Acquire the converted image frames and display them in real time; S7: Release decoding channel resources; The process of creating the compilation environment in step S1 is as follows: S11: Obtain the dynamic library files libacodec.lib, libavformat.lib, libavutil.lib, and libswscale.lib from ffmpeg and copy them to the working directory lib. S12: Copy the ffmpeg include header file to the include directory of the project; S13: Add the environment variable FFMPEGLIBPATH to the system and set this environment variable to the lib directory under the project directory; S14: Add an external library to the Qt project file. Add the library file from step S11, using the path indicated in step S13. In step S2, a new thread class named decode is created in the qt project directory. This class inherits from QThread. In step S2, the process of implementing the streaming initialization resource InitVideo in the thread class decode is as follows: S201: Use the library function avformat_network_init to initialize the network; S202: Use the library function avformat_alloc_context to create a streaming context object decoder_avfc; S203: Check if the context object decoder_avfc is empty. If it is empty, exit the process; otherwise, proceed to the next step. S204: Use the library function av_log_set_level to enable logging functionality; S205: Use the library function av_dict_set to set the avdic parameter of rstp; S206: Use the library function avformat_open_input to open the preset RTSP streaming URL; S207: Determine whether the result of step S206 is successful. If it is unsuccessful, record the error message and exit the process; otherwise, proceed to the next step. S208: Check if the parameter avdic is empty. If not, use the library function av_dict_free to release the object's resources; otherwise, proceed to the next step. S209: Use the library function avformat_find_stream_info to obtain the video stream. If the acquisition fails, use the library function avformat_close_input to close the context object decoder_avfc and release resources. Otherwise, proceed to the next step. In step S2, the process of decoding and initializing resources in the thread class decode is as follows: S211: Use the library function av_packet_alloc to create a packet object pkt. If creation is successful, proceed to the next step; otherwise, exit the process. S212: Use the library function avcodec_find_decoder to create a decoder codec. If creation is successful, proceed to the next step; otherwise, exit the process. S213: Use the library function av_parser_init to create a packet analysis object parser. If creation is successful, proceed to the next step; otherwise, exit the process. S214: Use the library function avcodec_alloc_context3 to create a decoding environment context object. If creation is successful, proceed to the next step; otherwise, exit the process. S215: Use the library function avcodec_oepn2 to open the decoder. If it fails, exit the process; otherwise, proceed to the next step. S216: Use the library function av_frame_alloc to create an image frame object frame. If it fails, exit the process; otherwise, proceed to the next step. In step S3, the process of creating the main pull stream thread is as follows: S31: Perform main loop for current pulling; S32: Check if the parameter setting flag is successful. If successful, proceed to the next step; otherwise, delay for 10µs. S33: Determine whether the streaming initialization step InitVideo in process S2 was successful. If it was, exit the connection establishment loop and continue to the next step; otherwise, continue the connection operation. S34: Enter the video stream acquisition loop; S35: Create a streaming data packet; S36: Initialize packet using the library function av_init_packet; S37: Set the packet's data pointer to a null pointer; S38: Set the packet's data size to 0; S39: Check if the streaming context decode_avfc is empty. If it is, return failure and exit the process; otherwise, proceed to the next step. S310: Use the library function av_read_frame to retrieve the streaming video data packets opened by decode_avfc. If opening fails, release the packet object; otherwise, store the data in the packet and proceed to the next step. S311: Determine if the stream index number of the packet object is the macro-defined video stream vdieo_Stream. If it is, proceed to the next decoding process and release the packet object; otherwise, release the packet object directly. In step S4, the process of decoding the data packet object obtained by the pull stream is as follows: S41: Use the library function avcode_send_packet to send the video stream data packet object (packet) obtained in process S3 to the decoder. If the sending fails, report an error and exit the process; otherwise, proceed to the next step. S42: Enter the loop to acquire decoded image frames; S43: Determine whether the result ret of the acquired image frame is greater than or equal to zero. If it is, proceed to the next step; otherwise, exit the loop process of acquiring image frames. S44: Use the library function avcodec_receive_frame to get the video frame where the context object dec_ctx is located. If the acquisition is successful, it is stored in the image frame object frame and the next step is performed. If it fails, an error message is printed and the process is exited. S45: Use the system function fflush to force the decoder to output image frames.

2. The real-time video decoding and playback method in the QT environment as described in claim 1, characterized in that, In step S5, the process of format conversion of the decoded image frame is as follows: S51: Use the library function av_frame_alloc to create an RGB image frame rgbFrame; S52: Use the library function avpicture_get_size to calculate the data size numBytes required to convert to rgb24 format; S53: Use the library function av_malloc to dynamically allocate an array rgbBuffer to store RGB image data, with a size of numBytes*sizeof(uint8_t); S54: Use the library function avpicture_fill to index the array of RGB image frames rgbFrame into rgbBuffer; S55: Check if the conversion context object image_convert_ctx is empty. If not, use the library function sws_freeContext to release the object; otherwise, proceed to the next step. S56: Use the library function sws_getContext to create a conversion context pair image_convert_ctx; S57: Determine if the conversion context pair image_convert_ctx is empty. If it is, exit the process; otherwise, proceed to the next step. S58: Use the library function sws_scale to convert the image frame and store the data in rgbFrame. Store the conversion result in result. S59: Determine if the conversion result is successful. If it is, proceed to the next step; otherwise, exit the process. S510: Send the converted image frame rgbBuffer to the interface.

3. The real-time video decoding and playback method in the QT environment as described in claim 2, characterized in that, In step S6, the process of acquiring the converted image frame and displaying it in real time is as follows: S61: Determine if the image frame object pointer is null. If it is, exit the process; otherwise, proceed to the next step. S62: Create an image display object, which is of type QImage; S63: Create a pixel map object pix, which is of type QPixmap; S64: Get the pointer to the display image object of the QLabel class, image_display; S65: Use the system function setPixmap to set the image object pointer image_display to display the image. The parameter is the return object of the pix from the image function.

4. The real-time video decoding and playback method in the QT environment as described in claim 3, characterized in that, In step S7, the process of releasing the decoding channel resources is as follows: S71: Use the library function av_parser_close to close the packet analysis object parser; S72: Use the library function avcodec_free_context to release the decoding context object; S73: Use the library function avformat_close_input to release the streaming context object decoder_avfc; S74: Use the library function av_packet_free to release the packet object; S75: Use the library function av_frame_free to release the image frame object; S76: Use the library function av_free to release the converted RGB array rgbBuffer.