Multifunctional short video system

Abstract

The embodiment of the invention discloses a multifunctional short video system which comprehensively uses an FFmpeg and an OpenGL (Open Graphics Library), and the FFmpeg is used for decoding and coding a video; the OpenGL is used for performing hardware rendering on the converted image to generate a new image, and the rendered new image is encoded and packaged to obtain a short video. Secondary development is carried out on the FFmpeg, an LUT algorithm is added, a video is decoded through the FFmpeg, the decoded image is converted through the LUT algorithm, rendering is carried out through OpenGL, and a new image is generated. According to the method, FFmpeg coding, decoding and packaging capabilities and OpenGL hardware rendering capability are comprehensively used, a short video image special effect is finally realized, and low consumption and real-time performance of video special effect processing are ensured; cross-platform open source multimedia is used, cross-platform use can be achieved, and cost is low; the filters are customized at the bottom layer, and the real-time superposition of the plurality of filters is realized through the filter chain.

Description

Technical Field

[0001] The present invention relates to the field of video editing. Specifically, it relates to a multifunctional short video system that can comprehensively utilize FFmpeg and OpenGL for video editing to improve the image special effects of short video processing. Background Art

[0002] With the advent of the 5G era, short videos have become an important way for mobile Internet users to obtain information. And users have more and more requirements for the production and editing of short videos. Short video editing systems in the prior art usually only include basic functions such as cropping and sticker display. Existing open-source software has many bugs and poor stability, and the final effects are uneven. It cannot be updated in a timely manner according to existing requirements, which makes it impossible to display some requirements in real time and there are certain lags during operation.

[0003] Since the current client system itself cannot meet the increasingly complex various short video requirements, introducing third-party SDKs has problems such as large package size, high cost, and slow demand iteration.

[0004] Therefore, how to improve the functions of the short video system, implement corresponding functions in a timely manner according to business requirements, define filters and implement the superposition of multiple filters has become a technical problem that urgently needs to be solved in the prior art. Summary of the Invention

[0005] In view of this, the embodiments of the present invention provide a multifunctional short video system that comprehensively utilizes FFmpeg and OpenGL for video editing, combines the encoding, decoding, and encapsulation capabilities of FFmpeg with the hardware rendering capabilities of OpenGL, and improves FFmpeg, thereby improving the encoding and decoding processing capabilities of the short video platform and enhancing the editing functions.

[0006] A multifunctional short video system, the system includes:

[0007] Comprehensively use FFmpeg and OpenGL. The FFmpeg is used for decoding and encoding of videos; the OpenGL is used for hardware rendering of the converted images to generate new images, and the newly rendered images are encoded and encapsulated to obtain short videos.

[0008] Optionally, the FFmpeg is redeveloped by adding a LUT algorithm. The video is decoded by FFmpeg, the decoded image is converted using the LUT algorithm, and then rendered using OpenGL to generate a new image.

[0009] Optionally, the short video system includes a recording module and an editing module;

[0010] The recording module and the editing module respectively call FFmpeg and OpenGL at the underlying level to achieve the underlying sharing of FFmpeg and OpenGL;

[0011] When using OpenGL for special effect display processing, data is processed in a chain, that is, each link on the chain is responsible for a special effect processing, and the generated new image is used as the input for the next link to continue processing. The special effect processing of image data is realized by hardware capabilities.

[0012] Optionally, the recording module is used to collect images using a camera, then preview them. During the preview, special effect processing can be performed using OpenGL. The special effect processing includes filters and beauty effects. After previewing the images, audio and video are synthesized, and then FFmpeg is used for video splicing.

[0013] Optionally, the audio and video synthesis includes using MediaMuxer to mix audio and video to generate a multimedia file, and using MediaCodec to compress and encode the audio and video;

[0014] Optionally, the video splicing uses the concat function of FFmpeg to merge multiple stream files or video files into one file in a specified order.

[0015] Optionally, the editing module is used to demux and decode a video using a player. For the decoded images, filter processing is performed using OpenGL, and for the processed images, editing is performed using FFmpeg. The editing includes cropping, stickers, and speed adjustment. After editing the video, it is encoded and encapsulated using FFmpeg again.

[0016] Optionally, the player uses FFmpeg to demux and decode a video, converts the decoded images using the LUT algorithm, and then performs filter operations using OpenGL to generate new images.

[0017] Optionally, both the recording module and the editing module include a preview function and a synthesis function.

[0018] The preview function is implemented using OpenGL. In the preview function, OpenGL can perform beauty effects and filters;

[0019] The synthesis function is implemented using FFmpeg. In the synthesis function, FFmgeg can perform speed adjustment, stickers, synthesis, and editing.

[0020] In summary, the present invention has the following advantages:

[0021] 1. By comprehensively using the encoding, decoding, and encapsulation capabilities of FFmpeg and the hardware rendering capabilities of OpenGL, the short video image special effects are finally realized, ensuring low consumption and real-time performance in video special effects processing.

[0022] 2. Both FFmpeg and OpenGL are cross-platform open-source multimedia, enabling the short video platform described in the present invention to be used across platforms with low cost, capable of rapid iteration to add new required functions, and enabling rapid integration according to the target platform subsequently.

[0023] 3. It can, according to the client characteristics and self-developed characteristics, realize the display of special effects using the system hardware capabilities at the upper layer, process special effects and synthesis at the bottom layer, and support cross-platform development.

[0024] 4. Custom filters are defined at the bottom layer, and real-time superposition of multiple filters is achieved through a filter chain. BRIEF DESCRIPTION OF THE DRAWINGS

[0025] Through the following description of the embodiments of the present invention with reference to the drawings, the above and other objects, features, and advantages of the present invention will become clearer. In the drawings:

[0026] Figure 1 is the overall structure diagram of the short video system according to a specific embodiment of the present invention;

[0027] Figure 2 is the functional logic diagram of the short video system according to a specific embodiment of the present invention.

[0028] 1. Recording module; 2. Editing module. DETAILED DESCRIPTION OF THE EMBODIMENTS

[0029] The following is a description of the present application based on embodiments, but the present application is not limited to these embodiments. In the following detailed description of the present application, some specific detailed parts are described in detail. Those skilled in the art can fully understand the present application without the description of these detailed parts. To avoid obscuring the essence of the present application, well-known methods, processes, procedures, elements, and circuits are not described in detail.

[0030] In addition, those of ordinary skill in the art should understand that the drawings provided herein are for illustrative purposes only, and the drawings are not necessarily drawn to scale.

[0031] Unless the context clearly requires otherwise, words such as "including" and "comprising" in the entire application document should be interpreted as having an inclusive meaning rather than an exclusive or exhaustive meaning; that is, it is the meaning of "including but not limited to".

[0032] In the description of this application, it should be understood that terms such as "first" and "second" are only used for descriptive purposes and cannot be construed as indicating or implying relative importance. In addition, in the description of this application, unless otherwise specified, the meaning of "a plurality" is two or more.

[0033] FFmpeg is a cross-platform open-source multimedia framework composed of a series of libraries and tools, capable of handling various audio and video formats, including operations such as encoding, decoding, transcoding, multiplexing, demultiplexing, and stream processing. With its high efficiency, flexibility, and rich functionality, FFmpeg has been widely used in the field of multimedia processing. FFmpeg contains multiple components and library files, supporting a variety of video codecs and audio codecs, including video encodings such as H.264, H.265 / HEVC, AV1, and audio encodings such as AAC, MP3, Opus, etc. In addition, FFmpeg also supports various container formats such as MP4, MKV, WebM, FLV, etc., as well as streaming media protocols such as RTMP, HLS, DASH, etc.

[0034] OpenGL (Open Graphics Library) is a cross-programming language and cross-platform graphics program interface, mainly used for the rendering of 2D and 3D graphics. It realizes the creation and operation of graphics through a series of function calls and is widely used in fields such as game development, computer graphics, virtual reality, and scientific visualization. OpenGL can run on different operating systems, utilize graphics acceleration hardware, provide high-performance graphics processing capabilities, and has a wide range of applications in image processing.

[0035] The multifunctional short video system of the present invention comprehensively uses FFmpeg and OpenGL. The FFmpeg is used for video decoding and encoding; the OpenGL is used for hardware rendering of the converted image to generate a new image, and the rendered new image is encoded and encapsulated to obtain a short video.

[0036] Therefore, the present invention comprehensively uses the encoding, decoding, and encapsulation capabilities of FFmpeg and the hardware rendering capabilities of OpenGL, and finally realizes the special effects of short video images.

[0037] Furthermore, the FFmpeg is further developed and the LUT (Lookup Table) algorithm is added. The video is decoded by FFmpeg, the decoded image is converted using the LUT algorithm, and then rendered using OpenGL to generate a new image.

[0038] Therefore, since the video decoding process uses the LUT lookup table algorithm, it not only reduces the amount of data processing but also improves the accuracy of image color processing, optimizes the visual effect of the video, and makes the post-production of video content more flexible and precise.

[0039] Further, referring to Figure 1 , a general structure diagram of a short video system according to a specific embodiment of the present invention is shown, including a recording module 1 and an editing module 2. The recording module and the editing module respectively call FFmpeg and OpenGL at the bottom layer to achieve the bottom-layer sharing of FFmpeg and OpenGL. When using OpenGL for special effect display processing, data is processed in a chain, that is, each link in the chain is responsible for one special effect processing, and the generated new image is used as the input for the next link to continue processing. The special effect processing of image data is implemented by using hardware capabilities to ensure low consumption and real-time performance of video special effect processing.

[0040] The recording module 1 is used to collect images using a camera, such as a camera in the Android system, and then perform preview. During the preview, special effect processing can be performed using OpenGL, and the special effect processing includes filters and beauty effects. After the previewed image is subjected to audio-video synthesis, FFmpeg is then used for video splicing.

[0041] In a specific embodiment, the audio-video synthesis includes using, for example, MediaMuxer in the Android system to mix audio and video to generate a multimedia file, and using MediaCodec to compress and encode the audio-video.

[0042] In another specific embodiment, the splicing of the video specifically uses the concat function of FFmpeg to merge multiple stream files or video files into one file in a specified order.

[0043] Therefore, the present invention can add elements such as stickers, watermarks, and filters to the recording screen when recording a video, implement functions such as whitening, skin smoothing, and blushing. The recorded video can control the display speed, support multi-segment recording splicing, deletion, and generate a media file that can be uploaded according to user configuration.

[0044] The editing module 2 is used to demux and decode the video using a player, perform filter processing on the decoded image using OpenGL, edit the processed image using FFmpeg, and the editing includes cropping, stickers, and speed change. The edited video is then encoded and encapsulated using FFmpeg.

[0045] In a specific embodiment, the player uses FFmpeg to demux and decode the video, converts the decoded image using the LUT algorithm, and then performs filter operations using OpenGL to generate a new image.

[0046] Therefore, the present invention can select local videos for operations such as cropping, adding stickers, watermarks, filters, and speed adjustment.

[0047] Further, referring to Figure 2 , both the recording module 1 and the editing module 2 include a preview function and a synthesis function.

[0048] The preview function includes real-time preview, effect preview, and material addition. Real-time preview allows users to see the playback effect of the video in real time during the editing process; effect preview enables users to see the changes in the effect in real time when adding filters, adjusting colors, or applying transition animations; adding materials to the video, the preview allows users to preview the effect of the materials. Through these preview functions, users can timely discover problems and make adjustments during the video production process to ensure that the quality and effect of the final work meet expectations.

[0049] Since OpenGL can render images in a hardware manner, the present invention uses OpenGL to be responsible for the implementation of preview-related functions, so as to see the final effect of image rendering in real time. And in the preview, video thumbnails can be quickly obtained at time intervals to achieve real-time display of frame-by-frame preview.

[0050] In the preview function, OpenGL can perform beauty effects and filters.

[0051] The video synthesis function includes synthesizing multiple segments of streaming media or video files, and encoding and encapsulating them according to the specified bit rate and format. Therefore, the present invention mainly uses FFmpeg to be responsible for the implementation of the synthesis function.

[0052] In the video synthesis function, FFmgeg can perform speed adjustment, add stickers, synthesize, and edit.

[0053] In summary, the present invention has the following advantages:

[0054] 1. By comprehensively using the encoding, decoding, and encapsulation capabilities of FFmpeg and the hardware rendering capabilities of OpenGL, the short video image special effects are finally realized, ensuring low consumption and real-time performance of video special effect processing.

[0055] 2. Both FFmpeg and OpenGL are cross-platform open-source multimedia, enabling the short video platform described in the present invention to be used across platforms, with low cost, capable of rapid iteration, adding new functions as needed, and being able to be quickly integrated according to the target platform in the future.

[0056] 3. It can, according to client characteristics and self-developed characteristics, achieve the display of special effects by utilizing the system hardware capabilities at the upper layer, process special effects and synthesis at the lower layer, and support cross-platform development.

[0057] 4. Custom filters at the lower layer can achieve real-time superposition of multiple filters through a filter chain.

[0058] The above are only the preferred embodiments of the present application and are not used to limit the present application. For those skilled in the art, various modifications and changes can be made to the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included within the protection scope of the present application.

Claims

1. A multi-functional short video system, characterized in that: FFmpeg and OpenGL are comprehensively used, where FFmpeg is used for video decoding and encoding; OpenGL is used for hardware rendering of the converted image to generate a new image, and the rendered new image is encoded and encapsulated to obtain a short video.

2. The short video system according to claim 1, characterized in that: FFmpeg is redeveloped by adding the LUT algorithm. The video is decoded by FFmpeg, the decoded image is converted using the LUT algorithm, and then rendered using OpenGL to generate a new image.

3. The short video system according to claim 2, characterized in that: The short video system includes a recording module and an editing module; The recording module and the editing module respectively call FFmpeg and OpenGL at the bottom layer to achieve the bottom-layer sharing of FFmpeg and OpenGL; When using OpenGL for special effect display processing, data is processed in a chain, that is, each link in the chain is responsible for a special effect processing, and the generated new image is used as the input for the next link to continue processing. The special effect processing of image data is implemented using hardware capabilities.

4. The short video system according to claim 3, characterized in that: The recording module is used to collect images using a camera, then preview, and can use OpenGL for special effect processing during the preview. The special effect processing includes filters and beauty effects. The previewed image is subjected to audio-video synthesis, and then the video is spliced using FFmpeg.

5. The short video system according to claim 4, characterized in that: The audio-video synthesis includes: using MediaMuxer to mix audio and video to generate a multimedia file, and using MediaCodec to compress and encode the audio-video.

6. The short video system according to claim 4, characterized in that: The splicing of the video is to use the concat function of FFmpeg to merge multiple stream files or video files into one file in a specified order.

7. The short video system according to claim 3, characterized in that: The editing module is used to demux and decode the video using a player, perform filter processing on the decoded image using OpenGL, edit the processed image using FFmpeg. The editing includes cropping, stickers, and speed adjustment, and then encode and encapsulate the edited video using FFmpeg.

8. The short video system according to claim 7, characterized in that: The player uses FFmpeg to demux and decode the video, converts the decoded image using the LUT algorithm, and then performs filter operations using OpenGL to generate a new image.

9. The short video system according to claim 3, characterized in that: Both the recording module and the editing module include a preview function and a synthesis function.

10. The short video system according to claim 9, characterized in that: The preview function is implemented using OpenGL. In the preview function, OpenGL can perform beauty enhancement and filtering. The synthesis function is implemented using FFmpeg. In the synthesis function, FFmpeg can perform speed adjustment, add stickers, synthesize, and edit.

Images