An openharmony-oriented end-side speech recognition optimization system

Abstract

This invention relates to an edge-side speech recognition optimization system for OpenHarmony, belonging to the field of speech recognition. The system includes a bidirectional information interaction layer (ArkTS), a cross-language N-API call layer, and a native inference execution layer. The ArkTS layer segments real-time recordings according to a preset time step and submits them asynchronously. The N-API layer achieves automatic resource release through `napi_wrap` binding and ensures safe string transmission through explicit allocation of independent buffers. The native layer receives incremental audio slices, performs context-dependent incremental inference, and returns the full recognition result by enforcing a unified floating-point calling convention to align the binary interface. This invention achieves stable operation and near real-time interactive experience for local speech recognition in the OpenHarmony environment.

Description

Technical Field

[0001] This invention relates to the field of speech recognition, and in particular to an edge-side speech recognition optimization system for OpenHarmony. Background Technology

[0002] With the widespread adoption of smart terminal devices, speech recognition, as an important human-computer interaction method, is widely used in mobile applications, smart homes, in-vehicle systems, and various edge devices. To reduce network dependence and protect user privacy, an increasing number of speech recognition solutions are adopting local speech recognition methods, meaning that the speech recognition model and inference process are completed locally on the terminal device.

[0003] As an open-source operating system for various types of terminal devices, OpenHarmony's operating environment varies significantly in terms of system capabilities, runtime interface support, and computing resources. Existing speech recognition solutions often struggle to run stably directly, and the following problems are commonly encountered during the porting process: 1. Binary Interface (ABI) Inconsistency Issues OpenHarmony uses a cross-compilation toolchain based on LLVM. Its floating-point calling convention conflicts with third-party pre-compiled inference libraries (such as ONNX Runtime) under certain architectures (e.g., differences between Soft-float and Hard-float modes). This leads to abnormal inference calculation results when calling across libraries, and in severe cases, identification logic errors or garbled output may occur.

[0004] 2. Lack of mechanisms for cross-language object lifecycle management In OpenHarmony, the ArkTS layer and the Native layer typically interact through the N-API interface. However, in some mainstream system versions (such as OpenHarmony 4.0), due to incomplete implementations or compatibility differences of some standard N-API interfaces (such as the interface used to bind object destruction callbacks), Native inference resources cannot keep pace with the lifecycle of ArkTS objects, which can easily lead to memory leaks or problems with resources not being released in a timely manner.

[0005] 3. Memory safety risks arising from string interaction When passing string parameters between ArkTS and the Native layer, some systems (such as OpenHarmony 4.0) have compatibility issues with the implementation of the Node-API string interface. When using the existing high-level encapsulation interface to obtain UTF-8 strings, illegal memory access will be triggered, causing the application to exit abnormally.

[0006] 4. Limited computing power on the device side leads to a decline in the user experience. OpenHarmony endpoint devices (such as IoT devices and mobile terminals) have limited CPU computing power. When performing non-streaming inference, directly processing long-duration voice data will result in a long computation wait time, causing UI feedback to lag and severely impacting the user's perceived "near real-time" interactive experience.

[0007] Therefore, there is an urgent need for an edge-side speech recognition optimization system for OpenHarmony to solve the aforementioned problems of ABI compatibility, memory security, and real-time interaction. Summary of the Invention

[0008] To address the issues of ABI compatibility, memory security, and real-time interaction, this invention provides an edge-side speech recognition optimization system for OpenHarmony.

[0009] This invention provides an edge-side speech recognition optimization system for OpenHarmony. The system includes: an ArkTS application interaction layer with bidirectional information exchange, an N-API cross-language call layer, and a Native inference execution layer, wherein; The ArkTS application interaction layer is used to establish an audio acquisition buffer, perform logical and physical slicing of real-time recording according to a preset time step to obtain incremental audio slices, and use the system's asynchronous scheduling mechanism to sequentially submit the incremental audio slices to the Native inference execution layer through the N-API cross-language call layer, and listen to the full recognition text results fed back by the N-API cross-language call layer, and update the interface display using a full coverage strategy. The N-API cross-language calling layer is used to implement object mapping, memory communication and bidirectional data transfer between the ArkTS application interaction layer and the Native inference execution layer. By calling the napi_wrap interface to bind the JS object of the ArkTS application interaction layer and the pointer of the CbData structure created by the Native inference execution layer, the automatic closed-loop release of Native resources and asynchronous callback context is realized. Furthermore, by explicitly requesting an independent buffer and manually adding the terminator, a physically isolated safe buffer is established to realize the safe transfer of bidirectional strings. The Native inference execution layer is used to integrate the speech recognition model and the inference engine. It eliminates cross-library call conflicts by forcing a unified floating-point calling convention to align the binary interface. It receives incremental audio slices transmitted through the N-API cross-language calling layer and maintains the context state to perform incremental inference. It produces the cumulative recognition result from the speech start point to the current time point and returns the full recognition text result to the ArkTS application interaction layer through the N-API cross-language calling layer.

[0010] Optionally, the automatic closed-loop release includes: when the JS object of the ArkTS application interaction layer is garbage collected, the system automatically triggers the destructor callback registered by the napi_wrap interface to release the Native resources in the CbData structure.

[0011] Optionally, the explicit request for an independent buffer includes: The actual UTF-8 encoded length of the source string is obtained using the length detection interface. An independent buffer with a length equal to the actual encoded length plus 1 byte is allocated in the Native heap memory. The string content is written through staged data copying, and the buffer is manually padded with a null terminator \0 at the end.

[0012] Optionally, the step of eliminating cross-library call conflicts by enforcing a unified floating-point calling convention to align binary interfaces includes: During the cross-compilation phase, the floating-point calling conventions of the target system's sysroot and its associated heterogeneous libraries are identified and analyzed. Force all dynamic link libraries and application native plugins to use a unified floating-point calling convention for their binary interfaces; By unifying the way floating-point calling conventions are passed between registers and stacks, register state pollution during cross-library calls is eliminated.

[0013] Optionally, the forced unified floating-point calling convention is softfp mode, and the elimination of cross-library call conflicts includes eliminating register state pollution between the application's native plugin and the pre-compiled third-party inference library.

[0014] Optionally, the pre-compiled third-party inference library includes onnxruntime.so.

[0015] Optionally, the full coverage strategy includes the ArkTS application interaction layer directly replacing the content of the interface text control with the full recognition text received each time it receives the full recognition text result, without performing incremental string concatenation operations, in order to support the backward semantic correction feature of the ASR model.

[0016] Optionally, the preset time step is 0.2 seconds.

[0017] Optionally, the ArkTS application interaction layer, the N-API cross-language call layer, and the Native inference execution layer work together through the following steps; Step S1: The Native inference execution layer identifies the floating-point calling convention of the OpenHarmony target system sysroot and its associated heterogeneous libraries, and forces a unified floating-point calling convention to align the binary interface; Step S2: The Native inference execution layer creates a CbData structure and calls the napi_wrap interface to bind the JS object of the ArkTS application interaction layer to the CbData structure pointer; Step S3: The ArkTS application interaction layer establishes an audio acquisition buffer and performs logical physical slicing on the real-time recording according to a preset time step to obtain incremental audio slices; Step S4: The N-API cross-language call layer establishes a physically isolated safe buffer by explicitly requesting an independent buffer and manually adding the terminator, and then passes the incremental audio slice to the Native inference execution layer through the safe buffer; Step S5: The Native inference execution layer receives incremental audio slices and maintains the context state to perform incremental inference, producing the cumulative recognition result from the start point of the speech to the current time point; Step S6: The N-API cross-language call layer feeds back the full text recognition results to the ArkTS application interaction layer through a secure buffer; Step S7: The ArkTS application interaction layer updates the interface display using a full coverage strategy. Attached Figure Description

[0018] Figure 1 This is a system framework diagram of an edge speech recognition optimization system for OpenHarmony according to an embodiment of this application.

[0019] Figure 2 This is a flowchart of an edge speech recognition optimization system for OpenHarmony, according to an embodiment of this application. Detailed Implementation

[0020] The present invention will be further described in detail below with reference to the accompanying drawings.

[0021] This specific embodiment is merely an explanation of the present invention and is not intended to limit the invention. After reading this specification, those skilled in the art can make modifications to this embodiment without contributing any inventive step, but such modifications are protected by patent law as long as they fall within the scope of the claims of the present invention.

[0022] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0023] Figure 1This is a system framework diagram of an edge speech recognition optimization system for OpenHarmony, as described in an embodiment of this application. Figure 1 As shown: The present invention provides an edge-side speech recognition optimization system for OpenHarmony, comprising an ArkTS application interaction layer for bidirectional information exchange, an N-API cross-language calling layer, and a Native inference execution layer, wherein; The ArkTS application interaction layer is used to establish an audio acquisition buffer, perform logical and physical slicing of real-time recordings according to a preset time step to obtain incremental audio slices, and use the system's asynchronous scheduling mechanism to sequentially submit the incremental audio slices to the Native inference execution layer through the N-API cross-language call layer, and listen for the full recognition text results fed back by the N-API cross-language call layer, and update the interface display using a full coverage strategy. The N-API cross-language calling layer is used to implement object mapping, memory communication, and bidirectional data transfer between the ArkTS application interaction layer and the Native inference execution layer. By calling the napi_wrap interface to bind the JS object of the ArkTS application interaction layer and the pointer of the CbData structure created by the Native inference execution layer, it realizes the automatic closed-loop release of Native resources and asynchronous callback context. It also realizes the secure bidirectional string transfer by explicitly requesting an independent buffer and manually adding the terminator to establish a physically isolated safe buffer. The Native Inference Execution Layer is used to integrate the speech recognition model and inference engine. It eliminates cross-library call conflicts by enforcing a unified floating-point calling convention to align the binary interface. It receives incremental audio slices passed through the N-API cross-language calling layer and maintains the context state to perform incremental inference. It produces the cumulative recognition result from the speech start point to the current time point and returns the full recognition text result to the ArkTS application interaction layer through the N-API cross-language calling layer.

[0024] Figure 2 This is a flowchart of an edge speech recognition optimization system for OpenHarmony according to an embodiment of this application, as shown below. Figure 2 As shown: The ArkTS application interaction layer, N-API cross-language calling layer, and Native inference execution layer work together through the following steps: Step S1: The Native inference execution layer identifies the floating-point calling convention of the OpenHarmony target system sysroot and its associated heterogeneous libraries, and forces a unified floating-point calling convention to align the binary interface.

[0025] Specifically, during the cross-compilation phase, the floating-point calling conventions of the target system's sysroot and its associated heterogeneous libraries are identified and analyzed. All dynamic link libraries and application native plugins' binary interfaces are forced to conform to a unified floating-point calling convention. By unifying the way floating-point calling conventions are passed between registers and the stack, register state pollution during cross-library calls is eliminated, ensuring the accurate consistency of inference tensor calculation results. It's important to note that forcing a unified floating-point calling convention in softfp mode, and eliminating cross-library call conflicts, includes eliminating register state pollution between application native plugins and pre-compiled third-party inference libraries (onnxruntime.so).

[0026] Step S1 resolves the conflict between the pre-compiled third-party library (onnxruntime.so) and the application's native module in the OpenHarmony environment regarding floating-point register usage rules.

[0027] Step S2: The Native inference execution layer creates a CbData structure and calls the napi_wrap interface to bind the JS object of the ArkTS application interaction layer to the CbData structure pointer.

[0028] Specifically, a structure CbData is created in the Native inference execution layer to encapsulate the inference instance and context; the napi_wrap interface is called to associate and bind the ArkTS layer JS object with the CbData structure pointer.

[0029] The explicit resource release logic of the Native inference execution layer is registered to the destructor callback parameter of the `napi_wrap` interface, thereby injecting the lifecycle observation of the underlying C++ objects into the system's garbage collection (GC) observation chain. When the JS objects in the ArkTS application interaction layer are garbage collected, the system automatically triggers the destructor callback registered in the `napi_wrap` interface, realizing the automatic closed-loop release of Native resources. When the ArkTS application interaction layer and the Native inference execution layer frequently interact with each other via text, this avoids the instability of high-level encapsulated interfaces and fundamentally eliminates the risk of memory leaks.

[0030] Step S3: The ArkTS application interaction layer establishes an audio acquisition buffer and performs logical physical slicing on the real-time recording according to a preset time step to obtain incremental audio slices.

[0031] Step S4: The N-API cross-language call layer establishes a physically isolated safe buffer by explicitly requesting an independent buffer and manually adding terminators, and then passes the incremental audio slices to the Native inference execution layer through the safe buffer.

[0032] Specifically, the actual UTF-8 encoded length of the ArkTS application interaction layer string is obtained using the length detection interface; an independent buffer space of length + 1 byte is explicitly allocated in the native heap memory; and a physically isolated safe buffer is established for the transmission of inference results by copying data in stages and manually padding the end with a null terminator \0, to prevent crashes caused by illegal memory access under the OpenHarmony memory mapping mechanism.

[0033] Step S5: The Native inference execution layer receives incremental audio slices and maintains the context state to perform incremental inference, producing the cumulative recognition result from the start point of the speech to the current time point.

[0034] Specifically, the inference engine in the Native inference execution layer maintains the context state of the current recognition session. Whenever a 0.2-second incremental audio slice is received, the inference engine performs incremental inference based on the existing state and updates the internal recognition sequence.

[0035] Step S6: The N-API cross-language call layer feeds back the full text recognition results to the ArkTS application interaction layer through a secure buffer.

[0036] Specifically, after completing incremental inference, the Native inference execution layer returns the full recognition text results from the start of the speech to the current time point to the ArkTS application interaction layer through the N-API callback interface.

[0037] Step S7: The ArkTS application interaction layer updates the interface display using a full coverage strategy.

[0038] Specifically, after receiving the full text recognition results, the ArkTS application interaction layer does not perform complex string concatenation logic, but directly updates the interface display using a full coverage strategy.

[0039] The full coverage strategy involves the ArkTS application interaction layer directly replacing the content of the interface text control with the full recognition text received each time it receives the full recognition text result, without performing incremental string concatenation operations, in order to support the backward semantic correction feature of the ASR (Automatic Speech Recognition) model.

[0040] The present invention has the following beneficial technical effects: 1. Resolved the computational distortion problem caused by ABI inconsistency: By enforcing a unified floating-point calling convention (softfp mode), register state pollution between heterogeneous pre-compiled libraries and application native modules in the OpenHarmony environment was eliminated, ensuring the accuracy and stability of ASR inference.

[0041] 2. Eliminates the risk of cross-language memory leaks: By using the destructor callback mechanism of napi_wrap to establish lifecycle shadow binding, automatic closed-loop management of native resources is achieved in versions such as OpenHarmony 4.0 that lack standard destructor interfaces.

[0042] 3. Eliminates memory safety risks associated with string interaction: Employs an explicit buffer strategy of "length probing + heap memory isolation + manual terminator" to completely bypass the unstable Node-API high-level interface, establish a physically isolated secure communication channel, and prevent system crashes caused by illegal memory access.

[0043] 4. Achieved near real-time interactive experience for edge-side non-streaming models: Through the architecture design of "post-recording incremental slicing + full-volume cumulative feedback + dynamic correction display", the streaming recognition experience is simulated on edge devices with limited computing power, which not only ensures recognition accuracy (by utilizing the complete context) but also provides low-latency visual feedback.

[0044] 5. Excellent system compatibility and portability: This solution provides a reliable alternative implementation to address the compatibility deficiencies of specific versions of OpenHarmony, enabling the speech recognition function to run stably on various terminal devices of the system, filling a technological gap in this field.

[0045] The above description is merely a preferred embodiment of the present invention and an explanation of the technical principles employed. Those skilled in the art should understand that the scope of disclosure in this invention is not limited to technical solutions formed by specific combinations of the above-described technical features, but should also cover other technical solutions formed by arbitrary combinations of the above-described technical features or their equivalents without departing from the foregoing disclosed concept. For example, technical solutions formed by substituting the above features with (but not limited to) technical features with similar functions disclosed in this invention.

Claims

1. An edge-side speech recognition optimization system for OpenHarmony, characterized in that, It includes the ArkTS application interaction layer with bidirectional information exchange, the N-API cross-language call layer, and the Native inference execution layer, among which; The ArkTS application interaction layer is used to establish an audio acquisition buffer, perform logical and physical slicing of real-time recording according to a preset time step to obtain incremental audio slices, and use the system's asynchronous scheduling mechanism to sequentially submit the incremental audio slices to the Native inference execution layer through the N-API cross-language call layer, and listen to the full recognition text results fed back by the N-API cross-language call layer, and update the interface display using a full coverage strategy. The N-API cross-language calling layer is used to implement object mapping, memory communication and bidirectional data transfer between the ArkTS application interaction layer and the Native inference execution layer. By calling the napi_wrap interface to bind the JS object of the ArkTS application interaction layer and the pointer of the CbData structure created by the Native inference execution layer, the automatic closed-loop release of Native resources and asynchronous callback context is realized. Furthermore, by explicitly requesting an independent buffer and manually adding the terminator, a physically isolated safe buffer is established to realize the safe transfer of bidirectional strings. The Native inference execution layer is used to integrate the speech recognition model and the inference engine. It eliminates cross-library call conflicts by forcing a unified floating-point calling convention to align the binary interface. It receives incremental audio slices transmitted through the N-API cross-language calling layer and maintains the context state to perform incremental inference. It produces the cumulative recognition result from the speech start point to the current time point and returns the full recognition text result to the ArkTS application interaction layer through the N-API cross-language calling layer.

2. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The automatic closed-loop release includes: when the JS object of the ArkTS application interaction layer is garbage collected, the system automatically triggers the destructor callback registered by the napi_wrap interface to release the Native resources in the CbData structure.

3. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The explicitly requested independent buffer includes: The actual UTF-8 encoded length of the source string is obtained using the length detection interface. An independent buffer with a length equal to the actual encoded length plus 1 byte is allocated in the Native heap memory. The string content is written through staged data copying, and the buffer is manually padded with a null terminator \0 at the end.

4. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The method of eliminating cross-library call conflicts by enforcing a unified floating-point calling convention to align binary interfaces includes: During the cross-compilation phase, the floating-point calling conventions of the target system's sysroot and its associated heterogeneous libraries are identified and analyzed. Force all dynamic link libraries and application native plugins to use a unified floating-point calling convention for their binary interfaces; By unifying the way floating-point calling conventions are passed between registers and stacks, register state pollution during cross-library calls is eliminated.

5. The edge-side speech recognition optimization system for OpenHarmony according to claim 4, characterized in that, The mandatory unified floating-point calling convention is softfp mode, and the elimination of cross-library call conflicts includes eliminating register state pollution between the application's native plugin and the pre-compiled third-party inference library.

6. The edge-side speech recognition optimization system for OpenHarmony according to claim 5, characterized in that, The pre-compiled third-party inference library includes onnxruntime.so.

7. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The full coverage strategy includes the ArkTS application interaction layer directly replacing the content of the interface text control with the full recognition text received each time it receives the full recognition text result, without performing incremental string concatenation operations, in order to support the backward semantic correction feature of the ASR model.

8. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The preset time step is 0.2 seconds.

9. The edge-side speech recognition optimization system for OpenHarmony according to claim 1, characterized in that, The ArkTS application interaction layer, N-API cross-language call layer, and Native inference execution layer work together through the following steps; Step S1: The Native inference execution layer identifies the floating-point calling convention of the OpenHarmony target system sysroot and its associated heterogeneous libraries, and forces a unified floating-point calling convention to align the binary interface; Step S2: The Native inference execution layer creates a CbData structure and calls the napi_wrap interface to bind the JS object of the ArkTS application interaction layer to the CbData structure pointer; Step S3: The ArkTS application interaction layer establishes an audio acquisition buffer and performs logical physical slicing on the real-time recording according to a preset time step to obtain incremental audio slices; Step S4: The N-API cross-language call layer establishes a physically isolated safe buffer by explicitly requesting an independent buffer and manually adding the terminator, and then passes the incremental audio slice to the Native inference execution layer through the safe buffer; Step S5: The Native inference execution layer receives incremental audio slices and maintains the context state to perform incremental inference, producing the cumulative recognition result from the start point of the speech to the current time point; Step S6: The N-API cross-language call layer feeds back the full text recognition results to the ArkTS application interaction layer through a secure buffer; Step S7: The ArkTS application interaction layer updates the interface display using a full coverage strategy.

Images