An input mapping control method and system based on offline speech recognition

Abstract

The application discloses an input mapping control method and system based on offline speech recognition. The method converts user speech into text keywords through an offline speech recognition engine, matches the text keywords with a preconfigured mapping table to determine a target operation type (keyboard operation, mouse operation, gamepad operation, application program startup, preset text input, custom macro sequence execution or system shortcut control operation), and calls a corresponding virtual drive interface to inject an operation signal. When multiple direction inputs are involved, smooth diagonal movement is realized through shared state merging. The application is completely offline, supports multi-modal unified mapping and multi-state execution mode, and is suitable for game operation, daily system control and other scenes on desktop and mobile terminals.

Description

Technical Field

[0001] This invention relates to the field of computer input control technology, specifically to a mapping control method and system that converts user voice commands into operation signals for various virtual input devices such as keyboards, mice, and game controllers through offline voice recognition, applicable to desktop operating systems and mobile operating system environments. Background Technology

[0002] For users with limited hand mobility (such as those with spastic cerebral palsy), using traditional keyboards, mice, and gamepads for daily operations or gaming presents significant difficulties. Existing technical solutions have the following shortcomings:

[0003] (1) Online voice assistant solution: It relies on network connection, which poses a risk of privacy leakage. The response delay is usually more than 500ms, which cannot meet the requirements of high real-time scenarios (such as games), and does not support mapping voice commands to game controller buttons.

[0004] (2) Dedicated auxiliary hardware equipment: high cost, poor portability, requires additional installation and debugging, which is not conducive to promotion and application.

[0005] (3) Existing voice control solutions: rely heavily on cloud recognition and do not support offline operation; only support simple button mapping and do not support continuous directional control of the joystick; lack support for complex input modes such as press, release, and mutual exclusion. Summary of the Invention

[0006] The purpose of this invention is to provide a completely offline, multimodal, low-latency voice input mapping method to overcome the shortcomings of the prior art.

[0007] The technical solution of the present invention is as follows:

[0008] An input mapping control method based on offline speech recognition includes the following steps:

[0009] S1. Voice Acquisition and Preprocessing: The user's voice signal is acquired in real time through an audio input device and processed in blocks according to a preset sampling rate and data block size.

[0010] S2. Offline speech recognition: Input the segmented audio data into the offline speech recognition engine, obtain the recognition results, and perform deduplication on the text.

[0011] S3. Keyword Matching: Match the recognized text with a pre-configured keyword mapping table. The mapping table stores the correspondence between voice keywords and target operations. The target operations include virtual input operations (keyboard keys, mouse operations, game controller operations) and system operations (application startup, preset text input, execution of custom macro sequences, system shortcut controls, etc.). During matching, longer keywords are prioritized, and fuzzy pinyin matching is provided as a fault tolerance mechanism.

[0012] S4. Virtual Input Execution: Determine the target operation type (including but not limited to keyboard key presses, mouse operations, game controller operations, application startup, text input, macro sequence execution, and system shortcut control) based on the matching results, and call the corresponding virtual input driver interface or system operation interface to generate and inject operation signals.

[0013] S5. Multi-directional input merging: When multiple directional input operations exist simultaneously (such as multiple directions of the joystick), the directional components are merged through a shared state mechanism to generate a composite directional signal and then output it uniformly.

[0014] 2. The method as described in claim 1, wherein the virtual input driver interface in step (c) comprises at least one of the following:

[0015] (a) A keyboard input simulation module that injects key events through the operating system input interface;

[0016] (b) A mouse input simulation module that injects mouse events through the operating system input interface;

[0017] (c) Create a virtual game controller device through a virtual device driver and inject analog input signals into the game controller input simulation module.

[0018] Furthermore, the shared state merging method in step S5 specifically involves: maintaining a set of currently active directions using the joystick identifier as the key; starting a background merging thread to read all active directions from the shared state set at fixed intervals, calculating the merged component values, and uniformly calling the joystick interface for output; when operation in a certain direction stops, the direction is removed from the set, the background thread automatically updates the output, and the joystick automatically returns to center when all directions stop. This method enables multiple independently triggered voice direction commands to be naturally merged into a smooth diagonal movement.

[0019] Furthermore, the virtual input operation supports multiple execution modes: (a) single-trigger mode, where a preset operation is executed once after the keyword is recognized; (b) continuous press-and-hold mode, which continuously sends signals until a stop command is received; (c) conditional release mode, including timed release and keyword-triggered release; and (d) repetitive trigger mode, which sends signals cyclically at preset intervals. An emergency stop mechanism is also included, with a preset stop keyword that immediately stops all ongoing operations upon recognition.

[0020] Furthermore, in a mobile operating system environment, this is achieved through the following methods: (a) an offline speech recognition engine is embedded in the mobile application to convert speech into text keywords; (b) the mapping operation is converted into virtual touch events (such as click, swipe, long press) and injected into the system input stack through the system accessibility service interface (AccessibilityService); (c) in a system environment that requires process isolation configuration, the accessibility service is configured to run as an independent process, and the instruction transmission between the main program and the accessibility service is realized through cross-process communication mechanisms (such as Intent).

[0021] The present invention has the following beneficial effects:

[0022] (1) Runs completely offline: No network required, protects privacy, and eliminates network latency.

[0023] (2) Multimodal unified mapping: The same set of voice commands supports three outputs: keyboard, mouse and game controller.

[0024] (3) Natural diagonal movement: Smooth diagonal movement is achieved by merging shared states.

[0025] (4) Flexible execution mode: Supports multiple modes such as single, press and hold, conditional release, and repeat.

[0026] (5) Cross-platform adaptation: Adapt to devices with different system environments through process isolation scheme.

[0027] (6) Low latency: End-to-end latency is controlled within 150ms. Attached Figure Description

[0028] Figure 1 This is a diagram of the overall system architecture of the present invention.

[0029] Figure 2 This is a flowchart illustrating the merging process of the joystick's oblique movement in this invention.

[0030] Figure 3 This is a state diagram of the polymorphic execution mode of voice commands in this invention.

[0031] Figure 4 This is an architecture diagram of a desktop embodiment of the present invention.

[0032] Figure 5 This is an architecture diagram of a mobile terminal embodiment of the present invention. Detailed Implementation Example 1 (Desktop)

[0033] This embodiment runs on a Windows operating system. The backend is implemented in Python. Speech recognition uses the Vosk offline engine (Chinese small model approximately 68MB), the virtual keyboard is injected via the pynput library, the virtual mouse via the pydirectinput library (compatible with DirectInput scenarios), and the virtual gamepad via the vgamepad library in conjunction with the ViGEmBus kernel driver. The frontend uses HTML / CSS / JavaScript to render a native window interface using PyWebView, and the system tray is implemented using pystray. See [link to overall architecture] for details. Figure 4 . Example 2 (Mobile)

[0034] This embodiment applies to the Android operating system environment. The application embeds an offline speech recognition engine (such as VoskAndroid). After the recognition result determines the target operation through a keyword matching module, a virtual touch event is injected using the dispatchGesture interface of AccessibilityService. In system environments requiring process isolation, AccessibilityService is configured to run as an independent process (via the android:process attribute in AndroidManifest.xml). The main application sends operation commands to the independent process via the Intent mechanism. See the architecture section below. Figure 5 . Example 3 (Extended Application)

[0035] In addition to game operations, this invention can also be applied to: opening specified applications via voice, inputting preset text content, and executing custom operation macro sequences. Users can customize the correspondence between voice keywords and target operations, and configuration files can be imported and exported.

[0036] The above description is merely a preferred embodiment of the present invention and does not limit the patent scope of the present invention. Any equivalent modifications made based on the description and drawings of the present invention, or direct or indirect applications in other related technical fields, are similarly included within the patent protection scope of the present invention.

Claims

1. An input mapping control method based on offline speech recognition, characterized in that, Includes the following steps: (a) Acquire user voice signals through an audio input device, input the segmented audio data into an offline speech recognition engine, and obtain text recognition results; (b) Match the text recognition result with a pre-configured keyword mapping table, which stores the correspondence between voice keywords and target operations. The target operations include at least one of the following: keyboard operation, mouse operation, game controller operation, application startup operation, preset text input operation, custom macro sequence execution operation, and system shortcut control operation. (c) Determine the type and parameters of the target operation based on the matching results, call the corresponding virtual input driver interface or system operation interface, and generate and inject the operation signal; (d) When multiple directional input operations exist simultaneously, the directional components are merged by sharing a state set, the composite directional signal is calculated, and then output uniformly through the corresponding virtual drive interface.

2. The method as described in claim 1, characterized in that, The virtual input driver interface in step (c) includes at least one of the following: (a) A keyboard input simulation module that injects key events through the operating system input interface; (b) A mouse input simulation module that injects mouse events through the operating system input interface; (c) Create a virtual game controller device through a virtual device driver and inject analog input signals into the game controller input simulation module.

3. The method as described in claim 1, characterized in that, In step (b), keyword matching prioritizes longer keywords and provides fuzzy matching based on pinyin as a fault-tolerance mechanism.

4. The method as described in claim 1, characterized in that, The shared state merging in step (d) includes: maintaining the currently active direction set using the joystick identifier as the key; starting a background merging thread to read the direction set at fixed intervals and calculate the merged component value; removing the corresponding direction from the set and updating the output when the direction operation stops.

5. The method as described in claim 1, characterized in that, The virtual input operation supports multiple execution modes, including single trigger mode, continuous press mode, timed release mode, keyword trigger release mode, and repeated trigger mode.

6. The method as described in claim 1, characterized in that, The offline speech recognition engine dynamically constructs grammatical constraints based on the currently active keywords in its vocabulary and immediately resets the recognizer state after a successful match to reduce the waiting delay for continuous recognition.

7. The method as described in claim 1, characterized in that, In a mobile operating system environment, the mapping operation is converted into a virtual touch event and injected into the system input stack through the accessibility service interface; in a system environment that requires process isolation configuration, the accessibility service is configured to run as an independent process, and the mapping instruction transmission between the main application and the accessibility service is realized through the cross-process communication mechanism.

8. An input mapping control system based on offline speech recognition, characterized in that, include: (a) A voice acquisition module for acquiring user voice signals in real time via an audio input device; (b) An offline speech recognition module, used to convert the segmented audio data into text recognition results; (c) Keyword matching module, which stores the mapping relationship between voice keywords and target operations, wherein the target operations include at least one of the following: keyboard operation, mouse operation, game controller operation, application launch operation, preset text input operation, custom macro sequence execution operation, system shortcut control operation, used to match the text recognition results with the mapping relationship; (d) Virtual input routing module, used to determine the type of target operation based on the matching result and distribute the operation instruction to the corresponding virtual driver module or system operation interface; (e) At least one virtual drive module generates a virtual input signal corresponding to at least one of keyboard operation, mouse operation or gamepad operation according to the type of the target operation, or performs application startup, preset text input, custom macro sequence execution or system shortcut control operation through the system operation interface; (f) Direction merging module, used to merge directional components and output them uniformly when there are multiple directional input operations at the same time by sharing a state set.

Images