Audio and video processing method and system under Gawan system

Abstract

The invention discloses an audio and video processing method and system under a swan gap system, and the method comprises the following steps: initializing an upper layer assembly and a lower layer assembly, constructing a cross-language communication layer, and loading a library file; generating a standardized audio and video processing command, submitting a task to generate a unique identifier, and scheduling according to priority; parameters are transmitted through a cross-language communication layer, a callback mechanism is bound, a primary layer verifies a task state, and after an isolation and exception handling mechanism is constructed, a task is executed in a single-thread serial mode and the state is reported in real time; and after receiving the state, the upper layer triggers callback to execute service logic, and cancels single and batch tasks, synchronously cleans queues and releases resources through a management and control interface. According to the method, convenient cross-language communication between the upper layer and the FFmpeg original layer is realized, the adaptive integration cost is reduced, the process safety operation is ensured, the multi-task concurrent conflict is relieved, the task state full-process perception is realized, the method is adaptive to multiple audio and video processing scenes, and a complete technical scheme is provided for the application of audio and video processing in the swan.

Description

Technical Field

[0001] This invention relates to the field of audio and video processing technology, and in particular to an audio and video processing method and system under the HarmonyOS system. Background Technology

[0002] With the continuous development of the HarmonyOS ecosystem, audio and video processing has become one of the core functional requirements of HarmonyOS applications. FFmpeg, as a mature open-source audio and video processing tool, possesses powerful encoding / decoding, format conversion, and streaming media processing capabilities, and is widely used in various audio and video applications. However, directly integrating FFmpeg into HarmonyOS faces several technical challenges: Cross-language interaction barriers: HarmonyOS application layers are developed using ArkTS, while FFmpeg core code is native C / C++. The memory models and execution environments of the two differ significantly. Traditional NAPI development requires manual handling of complex logic such as parameter conversion, memory management, and callback binding, resulting in a cumbersome adaptation process and extremely low development efficiency. Significant process security risks: When the native FFmpeg fftools tool encounters execution exceptions, such as incorrect parameters or encoding / decoding failures, it calls the exit() function to directly terminate the process, causing HarmonyOS applications to crash and failing to meet the requirements of library call scenarios. Stability requirements; Multi-task concurrency conflicts: FFmpeg relies on a large number of global variables. When multiple audio and video processing tasks are executed concurrently by multiple threads, problems such as state pollution, data corruption, and abnormal execution results are prone to occur, making it difficult to meet the multi-task processing requirements of HarmonyOS applications; High adaptation costs: The native FFmpeg code is not optimized for the HarmonyOS system. A series of issues such as static library compilation, dependency library completion, and parameter format adaptation need to be resolved, making integration difficult and the development cycle long; Lack of task state awareness: The native FFmpeg lacks a state interaction mechanism with the HarmonyOS application layer. The application layer cannot obtain the task execution progress, success or failure status in real time, making it difficult to achieve a user-friendly interaction.

[0003] Therefore, there is an urgent need in this field for a technical solution that can enable convenient cross-language interaction between the HarmonyOS application layer and the FFmpeg native layer, ensure process execution security, avoid multi-task concurrency conflicts, reduce adaptation and integration costs, and support real-time awareness of task execution status. Summary of the Invention

[0004] In view of this, the purpose of this invention is to provide an audio and video processing method and system under the HarmonyOS system, so as to solve the problems of complex cross-language interaction between the HarmonyOS application layer and the FFmpeg native layer, easy crashes due to abnormal process execution, multi-task concurrency conflicts, high adaptation and integration costs, and lack of task status awareness in the prior art. It achieves the effects of convenient cross-language communication, safe process operation, effective multi-task execution, and low-threshold adaptation and integration, improves the development efficiency, system stability and task execution reliability of audio and video processing, and meets the needs of HarmonyOS applications in scenarios such as audio and video downloading, format conversion, transcoding, and streaming media processing.

[0005] The specific plan is as follows: Firstly, this application provides an audio and video processing method under the HarmonyOS system, the method mainly including the following steps: S10: Initialize the upper-layer task submission component, callback registration component, and lower-layer task queue and worker thread scheduling component; build a cross-language communication layer to support interaction between the upper layer and the native layer; complete the loading of core libraries and dependency libraries; and set up the basic runtime environment. S20: Generate standardized audio and video processing commands by combining audio and video processing scenarios and business needs, and fill in codec, output format and transmission protocol parameters through pre-compiled templates; S30: Submit the audio and video processing command through the standardized interface, generate a unique task identifier, store the task in the task management container and sort it according to the preset task priority, and wake up the blocked worker thread. S40: The unique task identifier and command parameters are encapsulated and transmitted based on the cross-language communication layer. After receiving the parameters, the underlying layer uses the unique task identifier to bind the callback mechanism to realize the association between the task status and the matching callback. S50: The native layer verifies whether the task is in an uncancelled or timed-out state, adopts a thread-local storage state isolation strategy, constructs a non-local jump-type exception handling mechanism, and initializes the task execution environment; S60: Executes audio and video processing tasks in a single-threaded serial mode, collects and reports execution progress, result status, error information and error codes in real time, and triggers a preset exit mechanism when an exception occurs; S70: The upper layer receives the task status reported by the native layer, switches to the corresponding thread to trigger the callback mechanism, and executes the matching business logic; S80: Configures a standardized task management interface, supports canceling individual tasks through a unique task identifier, and terminates batch tasks through the core component destruction interface. It cleans up the task queue and status records at the underlying level, and releases hardware encoders, memory, dependency libraries and task execution environment associated configuration resources.

[0006] Furthermore, in step S10, the cross-language communication layer is built based on a cross-language interaction framework, the core library is a static library for audio and video processing, and the dependent libraries include a streaming media transmission library and an encryption authentication library.

[0007] Furthermore, in step S20, the standardized audio and video processing commands are generated by encapsulating APIs or chaining builder. In the case of hardware encoding and decoding, the pre-compiled template is automatically bound to the hardware encoder adaptation parameters.

[0008] Furthermore, in step S30, the unique task identifier is a globally unique identifier, the task management container is a task mapping storage structure, the preset task priority is divided into levels according to business importance, and the worker thread is a single execution thread.

[0009] Furthermore, in step S40, the cross-language communication layer completes parameter encapsulation and transmission through the pre-registration interface, and the binding of the callback mechanism is achieved through a combination of a unique task identifier and a callback identifier.

[0010] Furthermore, in step S50, the thread-local storage is implemented through task-level state isolation technology, the non-local jump exception handling mechanism is built based on non-local jump technology, and the task execution environment includes the adapted audio and video processing tool execution environment and encoder loading environment.

[0011] Furthermore, in step S60, the single working thread adopts a blocking wait, task wake-up, and serial execution operation mechanism. The audio and video processing task is executed through the adapted audio and video processing tool. In case of an exception, a preset jump mechanism is triggered to return an error flag.

[0012] Furthermore, in step S70, the switching to the corresponding thread is performed by switching to the application's main thread through a thread scheduling mechanism, and the matched business logic includes interface updates, user prompts, and operation records.

[0013] Furthermore, in step S80, the task management interface includes a single task cancellation interface, a batch task termination interface, and a core component destruction interface; the operations of clearing the task queue and status records and releasing resources are automatically triggered based on the status of task completion, task cancellation, or core component destruction.

[0014] Secondly, this application provides an audio and video processing system under the HarmonyOS system, which is used to implement the audio and video processing method under the HarmonyOS system as described above. The system includes an environment initialization module, a command generation module, a task scheduling and submission module, a cross-language communication module, a native layer execution environment construction module, an audio and video processing execution module, an upper layer callback processing module, and a task management and resource cleanup module. The environment initialization module is used to initialize the upper-layer task submission component, callback registration component, and lower-layer task queue and worker thread scheduling component, build a cross-language communication layer, load the core library and dependency library, and prepare the basic running environment. The command generation module is used to generate standardized audio and video processing commands based on scenario and business requirements, and to fill in codec, output format and transmission protocol parameters through pre-compiled templates; The task scheduling and submission module is used to submit tasks through a standardized interface, generate unique task identifiers, store tasks in a management container and sort them by priority, and wake up blocked worker threads. The cross-language communication module is used to encapsulate and transmit task identifiers and command parameters, and bind a callback mechanism after receiving parameters to realize the association and matching between task status and callback. The native layer execution environment construction module is used to verify whether the task is in an uncancelled or timed-out state, adopts a thread-local storage isolation strategy, constructs a non-local jump-type exception handling mechanism, and initializes the task execution environment. The audio and video processing execution module is used to execute tasks in a single-threaded serial mode, collect and report execution progress, result status, error information and error codes in real time, and trigger a preset exit mechanism when an abnormality occurs. The upper-layer callback processing module is used to receive the task status, switch to the corresponding thread to trigger the callback mechanism, and execute the matching business logic. The task management resource cleanup module is used to provide a task management interface, cancel individual tasks, terminate batch tasks, clean up task queues and status records, and release hardware encoders, memory, dependency libraries, and task execution environment associated configuration resources.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: (1) By constructing a standardized cross-language communication layer, the automatic cross-layer transmission of parameters between upper and lower layers is realized, which solves the complex problem of cross-language interaction and achieves convenient communication between the upper layer of HarmonyOS and the native layer of FFmpeg.

[0016] (2) By using standardized interfaces and pre-compiled templates to automatically fill in parameters, the cost of adaptation and integration is reduced and the development efficiency of audio and video processing is improved.

[0017] (3) By using a non-local jump exception handling mechanism and a thread-local storage isolation strategy, process crashes and task state pollution are avoided, ensuring the safe operation of the process.

[0018] (4) By combining single-threaded serial execution with priority scheduling, the orderly execution of multiple tasks is realized; by binding callback mechanism with unique task identifier, the full status of tasks is collected and reported in real time, and the full process of task status is effectively perceived.

[0019] (5) It adapts to various audio and video processing business scenarios, completes resource cleanup and release through the task management interface, reduces resource leakage, ensures stable system operation, and provides a complete technical solution for HarmonyOS application audio and video processing. Attached Figure Description

[0020] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are only embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on the provided drawings without creative effort.

[0021] Figure 1 This invention provides a flowchart of an audio and video processing method under the HarmonyOS system. Figure 2 This is a structural block diagram of an audio and video processing system under the HarmonyOS system provided in an embodiment of the present invention. Detailed Implementation

[0022] The present invention will be further described in detail below with reference to a specific application embodiment and accompanying drawings. This embodiment is implemented based on the technical solution of the present invention, and provides detailed implementation methods and specific operation processes, but the scope of protection of the present invention is not limited to the following embodiment.

[0023] Example 1 Please see Figure 1 This invention provides a technical solution: an audio and video processing method under the HarmonyOS system. Specifically, it includes the following steps: S10: Initialize the upper-layer task submission and callback registration components and the lower-layer task queue and worker thread scheduling components, build a cross-language communication layer to support the interaction between the upper layer and the native layer, complete the loading of core libraries and dependency libraries, and build the basic runtime environment.

[0024] S101: Initialize the upper-layer task submission and callback registration components, import the core audio and video processing class, obtain the global singleton of FFmpegManager, define the task status callback interface, and build the basic framework for upper-layer task submission and callback reception.

[0025] S102: Initialize the underlying task queue and worker thread scheduling component, create a priority task queue and active task mapping table, start a single worker thread and call the blocking method to enter the waiting state, and complete the basic preparation for underlying task scheduling.

[0026] S103: Based on the AKI cross-language interaction framework, a cross-language communication layer is built to support the interaction between the upper-layer ArkTS and the native layer C++. Cross-language data transmission and status reporting interfaces are pre-registered to break down the communication barriers between the upper and lower layers.

[0027] Specifically, the core registration of the cross-language calling interface is completed in the native layer's napi_ffmpeg.cpp file, enabling the upper layer of HarmonyOS to support the calling of native FFmpeg functions. The core code is as follows: / / Native layer core registration code (napi_ffmpeg.cpp) JSBIND_ADDON(ffmpegutils) / / Registers the module, generating the corresponding libffmpegutils.so library. JSBIND_GLOBAL() { JSBIND_PFUNCTION(executeFFmpegCommandAPP); / / Core interface for asynchronous execution of audio and video tasks } Among them, JSBIND_ADDON completes module registration and generates a so library that can be called by the upper layer, and JSBIND_PFUNCTION deploys the core functions of audio and video processing to the AKI thread pool for asynchronous execution, returns a Promise object to connect with the upper layer callback mechanism, completes the pre-deployment of the core interface of the cross-language communication layer, and avoids blocking the upper layer ArkTS main thread.

[0028] S104: Load the static core library and dependency libraries for audio and video processing. The dependency libraries include streaming media transmission library and encryption authentication library. Complete the deployment and initialization of the library files and build the basic runtime environment for upper-layer calls to cross-layer communication and lower-layer execution.

[0029] S20: Generate standardized audio and video processing commands by combining audio and video processing scenarios and business needs, and fill in codec, output format, and transmission protocol parameters through pre-compiled templates.

[0030] S201: Determine the core requirements for command generation based on actual audio and video processing scenarios and business needs. Audio and video processing scenarios include zero-copy fast operation, high-performance operation of hardware encoding and decoding, network stream processing, and custom requirement scenarios.

[0031] S202: Generate raw audio and video processing command strings that meet business requirements by using the FFmpegFactory API or the FFmpegCommandBuilder chain builder.

[0032] S203: Calls the underlying pre-compiled command template to automatically fill in the three core parameters of codec, output format, and transmission protocol, completing the standardized conversion of the original command.

[0033] S204: Performs parameter validation and format optimization on standardized commands, automatically binds hardware encoder adaptation parameters to hardware encoding and decoding scenarios, forming FFmpeg commands that can be directly executed at the underlying level, providing core data support for subsequent task submissions.

[0034] S30: Submit the audio and video processing command through the standardized interface, generate a unique task identifier, store the task in the task management container at the underlying level and sort it according to the preset task priority, and wake up the blocked worker thread.

[0035] S301: The upper layer calls the execute series of standardized interfaces, submits standardized audio and video processing commands, task timeout time and preset callback interfaces, and initiates a task execution request.

[0036] S302: The underlying layer generates a globally unique task identifier for this task and returns it to the upper layer for subsequent management and control, while also recording the core information of the task.

[0037] S303: The underlying layer stores tasks in a task management container, which is an active task mapping table. The task management container sorts the tasks to be executed in the task queue according to the preset priority based on business importance.

[0038] S304: The underlying method calls the thread wake-up method to wake up the blocked single worker thread, trigger the task execution process, and ensure that the core task enters the execution phase first.

[0039] S40: Based on the cross-language communication layer, a unique task identifier and command parameters are encapsulated and transmitted. After receiving the parameters, the underlying layer uses the unique task identifier to bind the callback mechanism to realize the association between the task status and the matching callback.

[0040] S401: With the help of the built cross-language communication layer, the upper-layer FFmpegManager automatically encapsulates core data such as task identifiers and standardized command parameters to form cross-language data packets.

[0041] Specifically, a globally unique task identifier is first generated in the FFMpegUtils.ets file at the ETS layer, and the corresponding callback function is bound to the global scope. The core code is as follows: / / ETS layer core registration callback code (FFMpegUtils.ets) let uuid = FFMpegUtils.generateUUID32(); / / Generate a globally unique task identifier / / Bind task callback functions, using UUIDs to distinguish different tasks libAddon.JSBind.bindFunction(uuid+"_onFFmpegProgress",options.onFFmpegProgress); libAddon.JSBind.bindFunction(uuid+"_onFFmpegSuccess",options.onFFmpegSuccess); Specifically, a unique identifier is generated using UUID to correspond to a single audio / video processing task. The progress, success, and other callback functions are bound to the global scope in the form of {UUID + function name}. This provides identification support for accurate callbacks in the subsequent native layer, achieving a one-to-one correspondence between tasks and callbacks and avoiding callback confusion.

[0042] / / Continuing from the previous section on ETS layer code import libAddon from 'libffmpegutils.so' / / Load the core module library generated by the Native layer registration / / Call the native asynchronous interface, passing the UUID and command parameters to complete cross-layer delivery. libAddon.executeFFmpegCommandAPP(uuid,options.cmds.length,options.cmds); Specifically, the import statement loads the .so library generated by the Native layer, enabling module integration between the upper layer and the native layer; the executeFFmpegCommandAPP interface takes in the UUID and standardized command parameters, and the AKI framework automatically completes the parameter mapping and cross-layer transmission, eliminating the need for manual NAPI parameter conversion and simplifying the cross-language interaction process.

[0043] S402: Through the cross-language communication layer pre-registration interface, data packets are passed from the upper layer ArkTS to the lower layer native C++, completing the cross-language delivery of parameters.

[0044] S403: The underlying layer receives and parses cross-language data packets, extracts task identifiers and command parameters, and submits the parameters to the Native layer task cache.

[0045] S404: By combining a unique task identifier and a callback identifier, the task callback mechanism is bound to achieve a precise association between the task execution status and the corresponding upper-level callback, ensuring that the status reporting is not off-target.

[0046] Specifically, in the native layer file napi_ffmpeg.cpp, the corresponding global callback function is obtained through the interface provided by the AKI framework based on the UUID passed from the upper layer. The core implementation logic is as follows: / / Core code for retrieving callbacks in the Native layer (napi_ffmpeg.cpp) / / Retrieve the upper-level bound JS callback function from the AKI framework using the callback name concatenated with the UUID: autoprogressCallback = JSBind::getFunction(uuid+"_onFFmpegProgress"); auto successCallback = JSBind::getFunction(uuid+"_onFFmpegSuccess"); The native layer receives the UUID passed from the upper layer and uses the combination of {UUID + function name} to accurately retrieve the corresponding upper-layer callback function from the AKI framework, thus completing the final binding between the task and the callback. When the task execution status changes, the retrieved callback function can be directly called to report the progress, result and other statuses to the upper layer ArkTS, realizing full-process feedback of the task status.

[0047] S50: The native layer verifies whether the task is in an uncancelled or timed-out state, adopts a thread-local storage state isolation strategy, constructs a non-local jump-type exception handling mechanism, and initializes the task execution environment.

[0048] S501: The native layer reads the current status of the task, verifies whether it is a compliant execution status that has not been canceled or timed out, and only performs subsequent processing on compliant tasks.

[0049] S502: The thread_local keyword is used to implement thread-local storage with task-level state isolation, allocating independent state storage space for the current task and avoiding state pollution between multiple tasks.

[0050] S503: Based on setjmp / longjmp nonlocal jump technology, it constructs a native nonlocal jump exception handling mechanism, presets error capture and jump points, and provides a foundation for graceful task exit.

[0051] S504: Loads the adapted audio and video processing tool execution environment and encoder loading environment, initializes the modified fftools execution environment, and completes the setup and verification of the task execution environment.

[0052] S60: Executes audio and video processing tasks in a single-threaded serial mode, collects and reports execution progress, result status, error information and error codes in real time, and triggers a preset exit mechanism in case of an exception.

[0053] S601: The underlying single-threaded worker adopts a serial execution mode, which retrieves compliant tasks from the priority task queue and starts the audio and video processing command execution process.

[0054] S602: Calls the modified fftools toolchain to execute standardized audio and video processing commands, and improves processing efficiency by utilizing GPU hardware resources in hardware encoding and decoding scenarios.

[0055] Specifically, to avoid native FFmpeg directly terminating the application process, the core functions of fftools were modified to use setjmp / longjmp to implement non-local jumps. The core code is as follows: / / 1. Modify exit_program() to replace the native exit with a non-local jump (fftools core modification) __thread volatile int longjmp_value; void exit_program(int ret) { longjmp_value = ret; longjmp(ex_buf__, ret); / / Exception jump, does not terminate the application process } / / 2. Set a jump point, execute the audio / video task, and gracefully return. int exe_ffmpeg_cmd(int argc, char **argv, Callbacks *callbacks) { int savedCode = setjmp(ex_buf__); / / Set the jump context if (savedCode == 0) { / / Normal execution: Calls the fftools core logic; exceptions trigger a redirect. if (ffmpeg_parse_options(argc, argv)<0 || transcode()<0) exit_program(1); } else { / / After redirection: Receive exception values ​​and prepare for a graceful return. main_return_code = longjmp_value; } return main_return_code; / / Only terminate the current task and return the result code. } The modified fftools toolchain replaces the native exit() logic with non-local jumps, and in hardware encoding and decoding scenarios, it can synchronously call GPU hardware resources to improve processing efficiency, which not only ensures task execution performance, but also avoids application process crashes caused by exceptions.

[0056] S603: Real-time collection of task execution progress, result status, detailed error information and standardized error codes to form a complete status reporting data packet.

[0057] S604: Asynchronously transmits status reporting data packets to the upper layer through the cross-language communication layer; triggers a preset exit mechanism in case of serious exceptions, terminating the execution of the current task without affecting the overall application process.

[0058] S70: The upper layer receives the task status reported by the native layer, switches to the corresponding thread to trigger the callback mechanism, and executes the matching business logic.

[0059] S701: The upper layer receives task status data packets reported by the native layer through the cross-language communication layer, and parses and extracts core content such as execution progress, result status, and error information.

[0060] S702: By using the HarmonyOS system's thread scheduling mechanism, it switches to the application's main thread, avoiding runtime exceptions caused by UI operations in child threads.

[0061] S703: Triggers the corresponding preset callback mechanism based on the task identifier to wake up the registered callback interface.

[0062] S704: Executes the matching business logic, including interface updates, user prompts, operation log recording, etc., and transforms the underlying execution results into actual business value.

[0063] S80: Configures a standardized task management interface, supports canceling individual tasks through a unique task identifier and terminating batch tasks through the core component destruction interface, cleans up the task queue and status records at the underlying level, and releases hardware encoders, memory, dependency libraries and task execution environment associated configuration resources.

[0064] S801: Provides standardized task management interfaces, including interfaces for canceling individual tasks and terminating batch tasks, supporting upper-level task management operations.

[0065] S802: Supports canceling a single pending or executing task by calling the interface through a unique task identifier, and terminating batch tasks through the core component destruction interface.

[0066] S803: The underlying synchronous cleanup task queue clears the target task, clears the task status record in the active task mapping table, and completes the cleanup of task-related data.

[0067] S804: Releases hardware encoder, occupied memory, dependent libraries and task execution environment associated configuration resources to avoid resource leaks and reserve a clean running environment for subsequent task execution.

[0068] Accordingly, see Figure 2 As shown, this application also provides an audio and video processing system under the HarmonyOS system, which is used to implement the audio and video processing method under the HarmonyOS system as described above. The system includes an environment initialization module, a command generation module, a task scheduling and submission module, a cross-language communication module, a native layer execution environment construction module, an audio and video processing execution module, an upper layer callback processing module, and a task management and resource cleanup module. The environment initialization module is used to initialize the upper-layer task submission component, callback registration component, and the lower-layer task queue and worker thread scheduling component, build a cross-language communication layer, load the core library and dependency library, and prepare the basic runtime environment. The command generation module is used to generate standardized audio and video processing commands based on scenario and business requirements, and fills in codec, output format and transmission protocol parameters through pre-compiled templates. The task scheduling and submission module is used to submit tasks through a standardized interface, generate unique task identifiers, store tasks in a management container and sort them by priority, and wake up blocked worker threads. The cross-language communication module is used to encapsulate and transmit task identifiers and command parameters. After receiving parameters, it binds a callback mechanism to achieve the association and matching between task status and callback. The native layer execution environment construction module is used to verify whether the task is in an uncancelled or timed-out state. It adopts a thread-local storage isolation strategy, builds a non-local jump-type exception handling mechanism, and initializes the task execution environment. The audio and video processing execution module is used to execute tasks in a single-threaded serial mode, collect and report execution progress, result status, error information and error codes in real time, and trigger a preset exit mechanism when an exception occurs. The upper-level callback processing module is used to receive the task status, switch to the corresponding thread to trigger the callback mechanism, and execute the matching business logic. The task management resource cleanup module provides a task management interface to cancel individual tasks, terminate batch tasks, clean up task queues and status records, and release hardware encoders, memory, dependency libraries, and task execution environment associated configuration resources.

[0069] In summary, this invention constructs a standardized cross-language communication layer, employs thread-local storage state isolation and non-local jump-type exception handling mechanisms, implements single-threaded serial execution and priority scheduling strategies, incorporates hardware encoding and decoding acceleration technology, and is equipped with a unified task management and resource cleanup mechanism. Combined with the collaborative operation of eight functional modules, it achieves closed-loop management of the entire audio and video processing process under the HarmonyOS system. This invention addresses the technical problems of complex cross-language interaction between the HarmonyOS application layer and the FFmpeg native layer, easy process crashes, multi-task concurrency conflicts, high adaptation and integration costs, and lack of task state awareness.

[0070] It should be noted that, in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another, and do not necessarily require or imply any such actual relationship or order between these entities or operations. The terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such process, method, article, or apparatus.

[0071] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An audio and video processing method under the HarmonyOS system, characterized in that, Includes the following steps: S10: Initialize the upper-layer task submission component, callback registration component, and lower-layer task queue and worker thread scheduling component; build a cross-language communication layer to support interaction between the upper layer and the native layer; complete the loading of core libraries and dependency libraries; and set up the basic runtime environment. S20: Generate standardized audio and video processing commands by combining audio and video processing scenarios and business needs, and fill in codec, output format and transmission protocol parameters through pre-compiled templates; S30: Submit the audio and video processing command through the standardized interface, generate a unique task identifier, store the task in the task management container and sort it according to the preset task priority, and wake up the blocked worker thread. S40: The unique task identifier and command parameters are encapsulated and transmitted based on the cross-language communication layer. After receiving the parameters, the underlying layer uses the unique task identifier to bind the callback mechanism to realize the association between the task status and the matching callback. S50: The native layer verifies whether the task is in an uncancelled or timed-out state, adopts a thread-local storage state isolation strategy, constructs a non-local jump-type exception handling mechanism, and initializes the task execution environment; S60: Executes audio and video processing tasks in a single-threaded serial mode, collects and reports execution progress, result status, error information and error codes in real time, and triggers a preset exit mechanism when an exception occurs; S70: The upper layer receives the task status reported by the native layer, switches to the corresponding thread to trigger the callback mechanism, and executes the matching business logic; S80: Configures a standardized task management interface, supports canceling individual tasks through a unique task identifier, and terminates batch tasks through the core component destruction interface. It cleans up the task queue and status records at the underlying level, and releases hardware encoders, memory, dependency libraries and task execution environment associated configuration resources.

2. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S10, the cross-language communication layer is built on a cross-language interaction framework, the core library is a static library for audio and video processing, and the dependent libraries include a streaming media transmission library and an encryption authentication library.

3. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S20, the standardized audio and video processing commands are generated by encapsulating APIs or chaining builder. In the case of hardware encoding and decoding, the pre-compiled template is automatically bound to the hardware encoder adaptation parameters.

4. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S30, the unique task identifier is a globally unique identifier, the task management container is a task mapping storage structure, the preset task priority is divided into levels according to business importance, and the worker thread is a single execution thread.

5. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S40, the cross-language communication layer completes parameter encapsulation and transmission through the pre-registration interface, and the binding of the callback mechanism is achieved through a combination of a unique task identifier and a callback identifier.

6. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S50, the thread-local storage is implemented through task-level state isolation technology, the non-local jump exception handling mechanism is built based on non-local jump technology, and the task execution environment includes the adapted audio and video processing tool execution environment and encoder loading environment.

7. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S60, the single working thread adopts a blocking wait, task wake-up, and serial execution operation mechanism. The audio and video processing task is executed by the adapted audio and video processing tool. In case of an exception, a preset jump mechanism is triggered to return an error flag.

8. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S70, the switching to the corresponding thread is performed by switching to the application's main thread through a thread scheduling mechanism. The matched business logic includes interface updates, user prompts, and operation records.

9. The audio and video processing method under the HarmonyOS system according to claim 1, characterized in that, In step S80, the task management interface includes a single task cancellation interface, a batch task termination interface, and a core component destruction interface; the operations of clearing the task queue and status records and releasing resources are automatically triggered based on the status of task completion, task cancellation, or core component destruction.

10. An audio and video processing system under the HarmonyOS system, used to implement the method as described in any one of claims 1 to 9, characterized in that, The system includes an environment initialization module, a command generation module, a task scheduling and submission module, a cross-language communication module, a native layer execution environment construction module, an audio and video processing execution module, an upper-layer callback processing module, and a task management and resource cleanup module. The environment initialization module is used to initialize the upper-layer task submission component, callback registration component, and lower-layer task queue and worker thread scheduling component, build a cross-language communication layer, load the core library and dependency library, and prepare the basic running environment. The command generation module is used to generate standardized audio and video processing commands based on scenario and business requirements, and to fill in codec, output format and transmission protocol parameters through pre-compiled templates; The task scheduling and submission module is used to submit tasks through a standardized interface, generate unique task identifiers, store tasks in a management container and sort them by priority, and wake up blocked worker threads. The cross-language communication module is used to encapsulate and transmit task identifiers and command parameters, and bind a callback mechanism after receiving parameters to realize the association and matching between task status and callback. The native layer execution environment construction module is used to verify whether the task is in an uncancelled or timed-out state, adopts a thread-local storage isolation strategy, constructs a non-local jump-type exception handling mechanism, and initializes the task execution environment. The audio and video processing execution module is used to execute tasks in a single-threaded serial mode, collect and report execution progress, result status, error information and error codes in real time, and trigger a preset exit mechanism when an abnormality occurs. The upper-layer callback processing module is used to receive the task status, switch to the corresponding thread to trigger the callback mechanism, and execute the matching business logic. The task management resource cleanup module is used to provide a task management interface, cancel individual tasks, terminate batch tasks, clean up task queues and status records, and release hardware encoders, memory, dependency libraries, and task execution environment associated configuration resources.

Images