Image overlay subtitle method for improving coding frame rate

Abstract

The invention provides an image overlay subtitle method for improving coding frame rate, comprising: creating a decoding buffer queue and at least two parallel threads, the parallel threads comprising a first thread and a second thread; the method comprises the following steps: receiving video stream data through a first thread, sending the data into a decoder for decoding, converting the data into data in a specific format, and storing the data in a decoding cache queue; and extracting the data from the decoding buffer queue through a second thread, superposing subtitles according to requirements, generating a bitmap corresponding to the subtitles, converting the data with the subtitles into an RGBA format for image synthesis, and converting the synthesized image into data in a specific format to be displayed by a display. According to the subtitle superposition method, the coding rate can be improved, and smooth playing of the subtitle superposition video on the embedded equipment is realized.

Description

Technical Field

[0001] The invention relates to the technical field of video pre- and post-processing, and in particular to an image superimposed subtitle method for improving coding frame rate. Background Art

[0002] As network transmission of high-definition videos becomes more and more common, it is very common to overlay subtitles on the transmitted videos. Currently, the usual way to overlay subtitles is to process the collected original video data and change the relevant pixels at the specific subtitle overlay position to the same color as the subtitle. If there is a need for subtitle transparency, the subtitle color is calculated by combining the original image pixel points and the corresponding overlay position with the calculated color. This method often requires traversing to query the subtitle overlay position and calculating the composite color. The process of traversing the subtitle area to modify pixels consumes a lot of CPU / GPU resources and takes a long time.

[0003] When it is necessary to superimpose subtitles on a video, the usual processing flow is to decode first, and the subtitle superposition is performed in sequence. After the decoding is completed, the subtitle superposition begins. When there is not much superimposed subtitle data, the frame rate can barely be maintained. For large subtitles and the need for additional transparency calculation, the calculation efficiency is obviously insufficient, especially on embedded devices with limited performance. Due to the limited encoding performance of the embedded device itself, the time it takes to encode a frame plus the time it takes to superimpose subtitles can easily cause the frame rate of the encoded video to fail to reach the nominal capacity, resulting in a significant reduction in the smoothness of the picture. Therefore, it is necessary to study an efficient image superimposition subtitle method that can be used on embedded devices with limited performance. Summary of the invention

[0004] The technical problem to be solved by the present invention is to provide a method for superimposing subtitles on an image with improved coding frame rate, so as to overcome the problem that superimposing subtitles on an embedded device leads to insufficient coding frame rate and poor video playback effect.

[0005] The problem of the present invention is achieved in this way: a method for superimposing subtitles on an image for improving the encoding frame rate comprises:

[0006] Creating a decoding cache queue and at least two parallel threads, wherein the parallel threads include a first thread and a second thread;

[0007] Receive video stream data through a first thread, send the data to a decoder for decoding, convert the data into data in a specific format and store it in a decoding cache queue;

[0008] The second thread takes data from the decoding cache queue, overlays subtitles as required, generates a bitmap corresponding to the subtitles, converts the data with subtitles into RGBA format for image synthesis, and converts the synthesized image into data in a specific format for display.

[0009] Furthermore, the specific format is set to a YUV format.

[0010] Furthermore, the method further comprises a to-be-encoded cache queue and a third thread; the to-be-encoded cache queue is used to cache the synthetic image data in a specific format of the second thread;

[0011] The frame data to be encoded is taken out from the to-be-encoded buffer queue by a third thread, and sent to the encoder for encoding, and the obtained video stream data with customized subtitles is stored or transmitted;

[0012] The third thread is executed in parallel with the first thread and the second thread.

[0013] Furthermore, the overall time of superimposing subtitles in the second thread is less than the frame sending interval in the first thread, which will not cause queue accumulation and the frame rate will not drop due to the time-consuming subtitle superimposition.

[0014] Furthermore, the pixel format conversion and image synthesis of the image are both processed by hardware acceleration of the chip.

[0015] Furthermore, the encoder and decoder are hardware encoder and hardware decoder respectively.

[0016] The advantages of the present invention are: by setting different parallel threads for calling the video decoding, subtitle synthesis and encoding processes respectively, the encoding time consumption is reduced, the problem of insufficient frame rate after superimposing subtitles on embedded devices can be effectively improved, and the picture fluency is improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] The present invention will be further described below in conjunction with embodiments with reference to the accompanying drawings.

[0018] Figure 1 The present invention is an execution flow chart of a method for superimposing subtitles on an image for improving the encoding frame rate.

[0019] Figure 2 It is a schematic diagram of the execution principle of three threads of the present invention. DETAILED DESCRIPTION

[0020] like Figure 1 and Figure 2 As shown, a method for superimposing subtitles on an image to improve the encoding frame rate of the present invention comprises:

[0021] Create a decoding cache queue and at least two parallel threads, wherein the parallel threads include a first thread Thread_1 and a second thread Thread_2;

[0022] Receive video stream data (such as H264 / MJPEG data) through the first thread, send the data to the decoder for decoding, convert it into data in a specific format and store it in a decoding cache queue;

[0023] The second thread takes data from the decoding cache queue, overlays subtitles as required, generates a bitmap corresponding to the subtitles, converts the data with subtitles into RGBA format for image synthesis, and converts the synthesized image into data in a specific format for display.

[0024] Preferably, the specific format is set to YUV format.

[0025] Preferably, the method further comprises a cache queue to be encoded and a third thread Thread_3; the cache queue to be encoded is used to cache the synthetic image data in a specific format of the second thread;

[0026] The frame data to be encoded is taken out from the to-be-encoded buffer queue by a third thread, and is sent to the encoder for encoding, and the obtained video stream data (such as H264) with customized subtitles is stored or transmitted;

[0027] The third thread is executed in parallel with the first thread and the second thread.

[0028] Preferably, the overall time of superimposing subtitles in the second thread is less than the frame sending interval in the first thread, so that queue accumulation will not be caused and the frame rate will not drop due to the time-consuming superimposition of subtitles.

[0029] Preferably, the pixel format conversion and image synthesis of the image are accelerated by chip hardware (chips of model RK3399 and RK3568 may be used), and the encoder and decoder respectively use hardware encoder and hardware decoder, which can reduce the burden on the processor and improve the processing efficiency of the embedded device.

[0030] In summary, the advantages of the present invention are as follows:

[0031] The present invention splits the decoding display, subtitle overlay and data re-encoding after overlay into different threads through multi-threaded operation, which may introduce some delay, but compared with the effect of frame rate reduction, the cost of this delay is negligible. The present invention can effectively improve the encoding frame rate of the embedded device after subtitles are overlaid. For example, the frame rate will be reduced to 25 frames after subtitles are overlaid, but now it can be maintained at 30 frames, making video playback smoother and improving user experience.

[0032] Although the specific implementation modes of the present invention are described above, those skilled in the art should understand that the specific implementation modes described are only illustrative and are not intended to limit the scope of the present invention. Equivalent modifications and changes made by those skilled in the art in accordance with the spirit of the present invention should be included in the scope of protection of the claims of the present invention.

Claims

1. A method for superimposing subtitles on an image to improve the encoding frame rate, characterized in that: include: Creating a decoding cache queue and at least two parallel threads, wherein the parallel threads include a first thread and a second thread; Receive video stream data through a first thread, send the data to a decoder for decoding, convert the data into data in a specific format and store it in a decoding cache queue; The second thread takes data from the decoding cache queue, overlays subtitles as required, generates a bitmap corresponding to the subtitles, converts the data with subtitles into RGBA format for image synthesis, and converts the synthesized image into data in a specific format for display.

2. The method for superimposing subtitles on an image to improve the encoding frame rate according to claim 1, characterized in that: The specific format is set to the YUV format.

3. The method for superimposing subtitles on an image to improve the encoding frame rate as claimed in claim 1, characterized in that: The method further includes a to-be-encoded buffer queue and a third thread; the to-be-encoded buffer queue is used to buffer the synthetic image data of a specific format of the second thread; The frame data to be encoded is taken out from the to-be-encoded buffer queue by a third thread, and sent to the encoder for encoding, and the obtained video stream data with customized subtitles is stored or transmitted; The third thread is executed in parallel with the first thread and the second thread.

4. The method for superimposing subtitles on an image to improve the encoding frame rate as claimed in claim 1, characterized in that: The overall time of superimposing subtitles in the second thread is shorter than the frame sending interval in the first thread, so queue accumulation will not be caused and the frame rate will not drop due to the time-consuming subtitle superimposition.

5. The method for superimposing subtitles on an image to improve the encoding frame rate as claimed in claim 1, characterized in that: The pixel format conversion and image synthesis of the image are both processed by hardware acceleration of the chip.

6. The method for superimposing subtitles on an image to improve the encoding frame rate as claimed in claim 1, characterized in that: The encoder and decoder respectively adopt a hardware encoder and a hardware decoder.

Images