Multi-version FFmpeg switching method and device and medium

Abstract

The embodiment of the invention discloses a multi-version FFmpeg switching method and device and a medium, and the method comprises the steps: after an upper-layer application initiates FFmpeg function call, intercepting the FFmpeg function call by utilizing an environment variable which points to a middle-layer library after modification; when it is intercepted that a video processing sub-function exists in the FFmpeg function, whether each process is in a video processing state or not is recorded through the created global thread local variable; and determining a video processing process according to the local variable, and executing video processing by using the sub-function of the video processing version FFmpeg function when the video processing sub-function is subsequently called by the process. The flexibility of video processing is ensured, seamless switching among versions is supported, and the overall applicability of the system is improved.

Description

Technical Field

[0001] The present invention relates to the technical field of RISC-V system multimedia processing, and in particular to a multi-version FFmpeg switching method, device and medium. Background Art

[0002] With the rapid development of RISC-V architecture, more and more systems and devices are beginning to adopt RISC-V processors. However, RISC-V systems face some compatibility challenges in multimedia processing, especially video decoding. Currently, many RISC-V systems rely on video processing units (VPUs) for video hardware decoding, and VPUs usually require a specific version of FFmpeg to work with them. As an open source multimedia processing tool, FFmpeg is widely used in audio and video encoding, decoding, conversion and other scenarios.

[0003] The existing solution requires that the FFmpeg version in the system strictly matches the hardware decoding plug-in, which makes it easy to have incompatibility issues when updating the FFmpeg version or different applications rely on different versions. Especially on the RISC-V architecture, the flexibility and adaptability of the system are limited, and users cannot freely switch FFmpeg versions when using hardware accelerated decoding, causing difficulties in development and deployment. Summary of the invention

[0004] The embodiments of the present invention provide a multi-version FFmpeg switching method, device and medium to solve the technical problem that the FFmpeg version cannot be freely switched in the RISC-V system in the prior art.

[0005] In a first aspect, an embodiment of the present invention provides a multi-version FFmpeg switching method, comprising: When the upper-layer application initiates a FFmpeg function call, the modified environment variable pointing to the middle-layer library is used to intercept the FFmpeg function call; When the video processing sub-function in the intercepted FFmpeg function is detected, the global thread local variable created is used to record whether each process is in the video processing state; The process of processing the video is determined according to the local variables, and when the process subsequently calls the video processing sub-function, the sub-function of the video processing version FFmpeg function is used to perform the video processing.

[0006] In a second aspect, an embodiment of the present invention further provides a multi-version FFmpeg switching device, comprising: The interception module is used to intercept the FFmpeg function call by using the modified environment variable pointing to the middle-layer library after the upper-layer application initiates the FFmpeg function call; The recording module is used to use the created global thread local variables to record whether each process is in the video processing state when the video processing sub-function in the FFmpeg function is intercepted; A process for processing a video is called according to a local variable, and when the process subsequently calls a video processing sub-function, the video processing is performed using a sub-function of the video processing version FFmpeg function.

[0007] In a third aspect, an embodiment of the present invention further provides a storage medium comprising computer executable instructions, which, when executed by a computer processor, are used to execute the multi-version FFmpeg switching method provided in the above embodiment.

[0008] The multi-version FFmpeg switching method, device and medium provided by the embodiment of the present invention intercept the FFmpeg function call by using the modified environment variable pointing to the middle layer library after the upper layer application initiates the FFmpeg function call; when the video processing sub-function is intercepted in the FFmpeg function, the global thread local variable created is used to record whether each process is in the video processing state; the process for processing the video is determined according to the local variable, and when the video processing sub-function is subsequently called by the process, the sub-function of the video processing version FFmpeg function is used to perform video processing. By using the newly added middle layer, a unified interface is defined, FFmepg function calls are intercepted, and thread local variables are used to enable each thread to record its own state to prevent interference during multi-threaded access. When the video processing function is subsequently called, it can be judged whether the current process is video processing. If it is video processing, the video processing version FFmepg is called, otherwise the system version FFmepg is called. Not only the flexibility of video processing is guaranteed, but also seamless switching between versions is supported, thereby improving the overall applicability of the system. BRIEF DESCRIPTION OF THE DRAWINGS

[0009] Other features, objects and advantages of the present invention will become more apparent from the detailed description of non-limiting embodiments made with reference to the following drawings: Figure 1 It is a flowchart of a multi-version FFmpeg switching method provided in Embodiment 1 of the present invention; Figure 2 It is a flowchart of a multi-version FFmpeg switching method provided in Embodiment 2 of the present invention; Figure 3 It is a structural diagram of a multi-version FFmpeg switching device provided in Embodiment 3 of the present invention. DETAILED DESCRIPTION

[0010] The present invention will be further described in detail below in conjunction with the accompanying drawings and embodiments. It is to be understood that the specific embodiments described herein are only used to explain the present invention, rather than to limit the present invention. It should also be noted that, for ease of description, only parts related to the present invention, rather than all structures, are shown in the accompanying drawings.

[0011] Embodiment 1 Figure 1 : is a flowchart of a multi-version FFmpeg switching method provided in Embodiment 1 of the present invention. This embodiment can be used for the situation where the FFmpeg version in the RISC-V system is freely switched according to the hard decoding plug-in. The method can be executed by a multi-version FFmpeg switching device, and specifically includes the following steps: Step 110, when the upper layer application initiates the FFmpeg function call, the FFmpeg function call is intercepted by using the modified environment variable pointing to the middle layer library.

[0012] In the prior art, video hardware decoding under the RISC-V architecture depends on a specific version of FFmpeg and its plug-ins. However, the FFmpeg version used by the desktop environment components in the system is often inconsistent with the FFmpeg version required for hard decoding, resulting in the hard decoding not working properly.

[0013] Since it is not possible to flexibly call the FFmpeg version that adapts to the hardware decoding requirements, the efficiency of video hardware decoding in the existing technology is often difficult to achieve the best. When the system cannot load the FFmpeg version that matches the VPU hardware, it will fall back to software decoding, resulting in slower decoding speed, increased resource consumption, and failure to fully utilize the performance advantages of hardware acceleration.

[0014] Although the problem can be partially solved by upgrading the overlay, it may destroy the compatibility with the video hard decoding plug-in.

[0015] Therefore, in this embodiment, a compatibility layer is first designed. Exemplarily, the compatibility layer may include: an intermediate layer, which may define a unified interface to intercept FFmepg function calls. In the upper application, that is, at the operating system level, when the application needs to call FFmepg for audio or video processing, the intermediate layer is used to intercept FFmpeg function calls. Optionally, the environment variable LD_PRELOAD= / usr / local / libmy_FFmpeg_intercept.so may be set, and written into / etc / profile to make the environment variable permanent, so that the intermediate layer library libmy_FFmpeg_intercept.so receives and intercepts the upper application's call to the FFmpeg function.

[0016] Step 120, when it is intercepted that there is a video processing sub-function in the FFmpeg function, the created global thread local variable is used to record whether each process is in the video processing state.

[0017] FFmpeg is an open source computer program that can be used to record and convert digital audio and video and convert them into streams. It includes a large number of sub-functions. The sub-functions are used to determine whether the video is currently being processed. For example, the video sub-functions such as avformat_open_input, avformat_find_stream_info, av_find_best_stream, avcodec_find_decoder, avcodec_open2, av_read_frame, avcodec_send_packet, avcodec_receive_frame, av_packet_unref, av_frame_free, avcodec_free_context, avformat_close_input can be used to determine whether the current state is to process the video.

[0018] Since there are multiple processes in the system, if all of them are intercepted, the processing of other processes will be affected. At the same time, since FFmpeg may be used by multiple threads, in this embodiment, an independent state is saved for each thread call, so that each thread can use a different FFmpeg version respectively without affecting the use of FFmpeg functions by other processes.

[0019] For example, a global thread local variable static__thread int is_video_processing may be created to record whether each thread is currently in the video processing state. After creation, the variable is initialized, that is, the variable of each thread is initially set to 0. It is also determined whether each thread calls a video processing sub-function. When the thread processes and calls a video processing sub-function, the variable of the corresponding thread is modified to 1.

[0020] Optionally, the determination of whether the thread calls a video processing sub-function may include: determining whether the thread calls a FFmpeg multimedia stream processing sub-function, and when calling, reading a url parameter of the FFmpeg multimedia stream processing sub-function, determining according to the url parameter whether the processing object of the FFmpeg multimedia stream processing sub-function is a processed file or whether the data format is a video, as well as video attributes, and using the url parameter to determine whether the thread calls a video processing sub-function, as well as the corresponding FFmpeg version.

[0021] The FFmpeg multimedia stream processing sub-function avformat_open_input is the initial function for video processing. Its function is to open the video to be processed and determine whether the incoming format is a video through its char *url parameter. If it is a video, set is_video_processing to 1.

[0022] Furthermore, when there are multiple versions, the video type of the char *url parameter can be used to determine the corresponding FFmpeg version to achieve adaptation to multiple FFmpeg versions. Furthermore, the most suitable FFmpeg version can be selected in combination with the application. Optionally, the application to which the process belongs is determined, the video processing format and decoding version of the application are searched, and the corresponding FFmpeg version is determined according to the video processing format, decoding version and char *url parameter.

[0023] Step 130, determining a process for processing the video according to the local variables, and when the process subsequently calls a video processing sub-function, performing video processing using a sub-function of the video processing version FFmpeg function.

[0024] For example, you can find a process with is_video_processing set to 1, and when it subsequently calls the video processing sub-function, use the sub-function of the video processing version FFmpeg function to perform video processing. This allows the video processing version FFmpeg to strictly match the hard decoding plug-in, improving video processing efficiency. When it is not a video, directly call the original function of the same name of the system version FFmpeg through dlsym.

[0025] In a preferred implementation of this embodiment, the method may further add the following steps: after the system version is installed normally, the deb package of the video processing version is decompressed, and the file path is modified to a path different from the system version; the decompressed deb package is repackaged into a deb package after the path is modified using the dpkg command and installed; all function interfaces of different versions of FFmpeg are redefined and declared in the middle layer library, with the same name as the original function interface of FFmpeg, for intercepting function calls. The above method allows multiple versions of FFmpeg to coexist. At the same time, all function interfaces of different versions of FFmpeg are redefined and declared in the middle layer library to facilitate the video processing process to call various versions of FFmpeg functions.

[0026] Embodiment 2 Figure 2It is a flowchart of the multi-version FFmpeg switching method provided by the second embodiment of the present invention. This embodiment is optimized based on the above embodiment. When the process subsequently calls the video processing sub-function, the sub-function of the video processing version FFmpeg function is used to perform video processing. The specific optimization is: calling the custom FFmpeg version opening function load_custom_FFmpeg() function to read the video processing version FFmpeg, and using the dynamic link function dlsym to call the video processing version FFmpeg function. In addition, the method can also add the following steps: when the process intercepts the FFmpeg multimedia stream cleanup and closing sub-function, the variables of the process in the global thread local variables are modified to the initial value. Call the custom FFmpeg version closing function unload_custom_FFmpeg() to close the video processing version FFmpeg, and use the dynamic link function dlsym to call the original function of the same name of the system version FFmpeg.

[0027] See also Figure 2 , the multi-version FFmpeg switching method comprises: Step 210, when the upper layer application initiates the FFmpeg function call, the FFmpeg function call is intercepted by using the modified environment variable pointing to the middle layer library.

[0028] Step 220, when the upper layer application initiates the FFmpeg function call, the FFmpeg function call is intercepted by using the modified environment variable pointing to the middle layer library.

[0029] Step 230, determine the process for processing the video according to the local variable, and when the process subsequently calls the video processing sub-function, call the custom open FFmpeg version function load_custom_FFmpeg() function to read the video processing version FFmpeg, and use the dynamic link function dlsym to call the function of the video processing version FFmpeg.

[0030] In this embodiment, a custom load_custom_FFmpeg() function may be provided in the middle layer to open the FFmpeg version specified by the process, and to call the function of the video processing version FFmpeg through the dynamic link function dlsym, and to call the function in the library corresponding to the video processing version FFmpeg using the dynamic link function dlsym.

[0031] Step 240, when the process intercepts the FFmpeg multimedia stream cleanup and closing sub-function, the variables of the process in the global thread local variables are modified to the initial values.

[0032] In this embodiment, avformat_close_input is a FFmpeg multimedia stream cleanup and closing sub-function, which can determine that the process has completed the current video processing. Therefore, the variables of the process in the global thread local variables can be modified to the initial value, that is, set to 0, so that it can call the original FFmpeg version.

[0033] Step 250, call the custom FFmpeg version closing function unload_custom_FFmpeg() to close the video processing version FFmpeg, and use the dynamic link function dlsym to call the original function of the same name of the system version FFmpeg.

[0034] For example, the video processing version FFmpeg can be closed by using the newly added FFmpeg version closing function unload_custom_FFmpeg() in the middle layer, and the dynamic link function dlsym can be used to call the original function of the same name of the system version FFmpeg.

[0035] This embodiment uses the sub-function of the video processing version FFmpeg function to perform video processing when the process subsequently calls the video processing sub-function. The specific optimization is: call the custom open FFmpeg version function load_custom_FFmpeg() function to read the video processing version FFmpeg, and use the dynamic link function dlsym to call the video processing version FFmpeg function. In addition, the method can also add the following steps: when the process intercepts the FFmpeg multimedia stream cleanup and close sub-function, modify the variables of the process in the global thread local variables to the initial value. Call the custom close FFmpeg version function unload_custom_FFmpeg() to close the video processing version FFmpeg, and use the dynamic link function dlsym to call the original function of the same name of the system version FFmpeg. Using the above method, video processing is performed using the sub-function that calls the video processing version FFmpeg function. The FFmpeg multimedia stream cleanup and shutdown sub-function is used to modify the process variables, and the video processing version of FFmpeg is closed by calling the custom FFmpeg version shutdown function unload_custom_FFmpeg() added in the middle layer, and the dynamic link function dlsym is used to call the original function of the same name of the system version of FFmpeg to avoid affecting the system version of FFmpeg.

[0036] Embodiment 3 Figure 3 is a schematic diagram of the structure of a multi-version FFmpeg switching device provided in Embodiment 3 of the present invention, see Figure 3 , the multi-version FFmpeg switching device comprises: The interception module 310 is used to intercept the FFmpeg function call by using the modified environment variable pointing to the middle layer library after the upper layer application initiates the FFmpeg function call; The recording module 320 is used to record whether each process is in the video processing state by using the created global thread local variable when a video processing sub-function is intercepted in the FFmpeg function; The execution module is used to determine the process of processing the video according to the local variables, and when the process subsequently calls the video processing sub-function, use the sub-function of the video processing version FFmpeg function to perform video processing.

[0037] The multi-version FFmpeg switching device provided in this embodiment intercepts the FFmpeg function call by using the modified environment variable pointing to the middle layer library after the upper layer application initiates the FFmpeg function call; when the video processing sub-function is intercepted in the FFmpeg function, the global thread local variable created is used to record whether each process is in the video processing state; the process for processing the video is determined according to the local variable, and when the video processing sub-function is subsequently called by the process, the sub-function of the video processing version FFmpeg function is used to perform video processing. Using the newly added middle layer, a unified interface is defined, FFmepg function calls are intercepted, and thread local variables are used to enable each thread to record its own status to prevent interference during multi-threaded access. When the video processing function is subsequently called, it can be judged whether it is currently video processing. If it is video processing, the video processing version FFmepg is called, otherwise the system version FFmepg is called. It not only ensures the flexibility of video processing, but also supports seamless switching between versions, thereby improving the overall applicability of the system.

[0038] Based on the above embodiments, the recording module includes: Create a unit to create a global thread local variable static__thread int is_video_processing; The initialization unit is used to initially set the variables of each thread to 0; The modification unit is used to determine whether the thread calls the video processing sub-function, and when the thread calls the video processing sub-function, the variable of the corresponding thread is modified to 1.

[0039] Based on the above embodiments, the modification unit is used to: Determine whether the thread calls the FFmpeg multimedia stream processing sub-function. When calling, read the url parameter of the FFmpeg multimedia stream processing sub-function, determine whether the processing object of the FFmpeg multimedia stream processing sub-function is a processed file or whether the data format is a video, and the video attributes according to the url parameter, and use the url parameter to determine whether the thread calls the video processing sub-function and the corresponding FFmpeg version.

[0040] Based on the above embodiments, the device further includes: The variable modification module is used to modify the variables of the process in the global thread local variables to initial values ​​when the process intercepts the FFmpeg multimedia stream cleanup and closing sub-function.

[0041] Based on the above embodiments, the device further includes: The shutdown module is used to call the custom FFmpeg version shutdown function unload_custom_FFmpeg() to shut down the video processing version FFmpeg, and use the dynamic link function dlsym to call the original function of the same name of the system version FFmpeg.

[0042] Based on the above embodiments, the execution module includes: The calling unit is used to call the custom open FFmpeg version function load_custom_FFmpeg() function to read the video processing version FFmpeg, and use the dynamic link function dlsym to call the function of the video processing version FFmpeg.

[0043] Based on the above embodiments, the interception module includes: The setting unit is used to set the environment variable LD_PRELOAD= / usr / local / libmy_FFmpeg_intercept.so and write it into / etc / profile to make the environment variable permanent. The middle layer library libmy_FFmpeg_intercept.so receives and intercepts the call of the upper layer application to the FFmpeg function.

[0044] Based on the above embodiments, the device further includes: The path modification module is used to decompress the deb package of the video processing version after the system version is installed normally, and modify the file path to a path different from that of the system version; Repacking module, which is used to repack the decompressed deb package into a deb package and install it after modifying the path using the dpkg command; Redefine the module, which is used to redefine and declare all function interfaces of different versions of FFmpeg in the middle-layer library. It has the same name as the original function interface of FFmpeg and is used to intercept function calls.

[0045] The multi-version FFmpeg switching device provided in the embodiment of the present invention can execute the multi-version FFmpeg switching method provided in any embodiment of the present invention, and has the corresponding functional modules and beneficial effects of the execution method.

[0046] Embodiment 4 Embodiment 4 of the present invention further provides a storage medium comprising computer executable instructions, wherein the computer executable instructions, when executed by a computer processor, are used to execute any of the multi-version FFmpeg switching methods provided in the above embodiments.

[0047] The computer storage medium of the embodiment of the present invention can adopt any combination of one or more computer-readable media. The computer-readable medium can be a computer-readable signal medium or a computer-readable storage medium. The computer-readable storage medium can be, for example, but not limited to, a system, device or device of electricity, magnetism, light, electromagnetic, infrared, or semiconductor, or any combination of the above. More specific examples (non-exhaustive list) of computer-readable storage media include: an electrical connection with one or more wires, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the above. In this document, a computer-readable storage medium can be any tangible medium containing or storing a program, which can be used by an instruction execution system, device or device or used in combination with it.

[0048] Computer-readable signal media may include data signals propagated in baseband or as part of a carrier wave, which carry computer-readable program code. Such propagated data signals may take a variety of forms, including but not limited to electromagnetic signals, optical signals, or any suitable combination of the above. Computer-readable signal media may also be any computer-readable medium other than a computer-readable storage medium, which may send, propagate, or transmit a program for use by or in conjunction with an instruction execution system, apparatus, or device.

[0049] The program code embodied on the computer readable medium may be transmitted using any appropriate medium, including but not limited to wireless, wireline, optical fiber cable, RF, etc., or any suitable combination of the foregoing.

[0050] Computer program code for performing the operations of the present invention may be written in one or more programming languages ​​or a combination thereof, including object-oriented programming languages, such as Java, Smalltalk, C++, and conventional procedural programming languages, such as "C" or similar programming languages. The program code may be executed entirely on the user's computer, partially on the user's computer, as a separate software package, partially on the user's computer and partially on a remote computer, or entirely on a remote computer or device. In the case of a remote computer, the remote computer may be connected to the user's computer through any type of network, including a local area network (LAN) or a wide area network (WAN), or may be connected to an external computer (e.g., via the Internet using an Internet service provider).

[0051] Note that the above are only preferred embodiments of the present invention and the technical principles used. Those skilled in the art will understand that the present invention is not limited to the specific embodiments described herein, and that various obvious changes, readjustments and substitutions can be made by those skilled in the art without departing from the scope of protection of the present invention. Therefore, although the present invention has been described in more detail through the above embodiments, the present invention is not limited to the above embodiments, and may include more other equivalent embodiments without departing from the concept of the present invention, and the scope of the present invention is determined by the scope of the appended claims.

Claims

1. A multi-version FFmpeg switching method, characterized in that: include: When the upper-layer application initiates a FFmpeg function call, the modified environment variable pointing to the middle-layer library is used to intercept the FFmpeg function call; When the video processing sub-function in the intercepted FFmpeg function is detected, the global thread local variable created is used to record whether each process is in the video processing state; The process of processing the video is determined according to the local variables, and when the process subsequently calls the video processing sub-function, the sub-function of the video processing version FFmpeg function is used to perform the video processing.

2. The method according to claim 1, characterized in that The global thread local variables created by using are used to record whether each process is in the video processing state, including: Create a global thread local variable static __thread int is_video_processing; Initialize each thread's variables to 0; Determine whether the thread calls the video processing sub-function. When the thread calls the video processing sub-function, modify the variable of the corresponding thread to 1.

3. The method according to claim 2, characterized in that The determining whether the thread calls the video processing sub-function comprises: Determine whether the thread calls the FFmpeg multimedia stream processing sub-function. When calling, read the url parameter of the FFmpeg multimedia stream processing sub-function, determine whether the processing object of the FFmpeg multimedia stream processing sub-function is a processed file or whether the data format is a video, and the video attributes according to the url parameter, and use the url parameter to determine whether the thread calls the video processing sub-function and the corresponding FFmpeg version.

4. The method according to claim 3, characterized in that The method further comprises: When the process intercepts the FFmpeg multimedia stream cleanup and closing sub-function, the variables of the process in the global thread local variables are modified to the initial values.

5. The method according to claim 4, characterized in that The method further comprises: Call the custom FFmpeg version closing function unload_custom_FFmpeg() to close the video processing version FFmpeg, and use the dynamic link function dlsym to call the original function of the same name of the system version FFmpeg.

6. The method according to claim 3, characterized in that: When the process subsequently calls the video processing sub-function, the sub-function of the video processing version FFmpeg function is used to perform video processing, including: Call the custom open FFmpeg version function load_custom_FFmpeg() function to read the video processing version FFmpeg, and use the dynamic link function dlsym to call the function of the video processing version FFmpeg.

7. The method according to claim 1, characterized in that The method of using the modified environment variable pointing to the middle layer library to intercept the FFmpeg function call includes: Set the environment variable LD_PRELOAD= / usr / local / libmy_FFmpeg_intercept.so and write it to / etc / profile to make the environment variable permanent. The middle-layer library libmy_FFmpeg_intercept.so receives and intercepts the calls to FFmpeg functions from the upper-layer applications.

8. The method according to claim 1, characterized in that The method further comprises: After the system version is installed normally, decompress the deb package of the video processing version and change the file path to a path different from that of the system version; Use the dpkg command to modify the path, unpack the deb package, repackage it into a deb package and install it; Redefine and declare all function interfaces of different versions of FFmpeg in the middle-layer library, with the same name as the original function interface of FFmpeg, for intercepting function calls.

9. A multi-version FFmpeg switching device, characterized in that: include: The interception module is used to intercept the FFmpeg function call by using the modified environment variable pointing to the middle-layer library after the upper-layer application initiates the FFmpeg function call; The recording module is used to use the created global thread local variables to record whether each process is in the video processing state when the video processing sub-function in the FFmpeg function is intercepted; The execution module is used to determine the process of processing the video according to the local variables, and when the process subsequently calls the video processing sub-function, use the sub-function of the video processing version FFmpeg function to perform video processing.

10. A storage medium containing computer executable instructions, characterized in that: The computer executable instructions, when executed by a computer processor, are used to execute the multi-version FFmpeg switching method as described in any one of claims 1-8.

Images