A voice-driven intelligent device global control system and method

Abstract

This invention provides a voice-driven global control system and method for intelligent devices, applicable to intelligent devices with a display screen, voice input component, touch input component, and operating system. The system recognizes user voice as text commands, combines foreground applications, active windows, accessibility node tree, input method status, and sensitive interface judgment information to generate candidate system-level input events, which are then injected through an accessibility service interface or input assistance interface. Through three-channel consistency authorization, security authorization matrix, one-time execution ticket, application operation recipe, and post-state verification, the system performs time-limited, domain-limited, single-time authorization, and execution result confirmation for events. Automatic access control is implemented on sensitive interfaces such as payment, password, transfer, and permission authorization.

Description

Technical Field

[0001] This invention relates to the field of human-computer interaction technology, specifically to a voice-driven global control system and method for intelligent devices. Background Technology

[0002] With the widespread adoption of smart devices such as smartphones, tablets, smart TVs, in-vehicle infotainment systems, smart cockpits, smartwatches, smart glasses, head-mounted displays, and smart home control centers, users have higher demands for human-computer interaction methods. Traditional smart devices primarily rely on touchscreens, remote controls, keyboards, mice, knobs, or physical buttons for operation. While these interaction methods can meet operational needs in general scenarios, they present significant inconveniences in situations where users' hands are unable to touch the screen, while driving, cooking, exercising, wearing gloves, being far from the display device, or having limited physical mobility.

[0003] Voice interaction technology uses a microphone to capture user voice and a speech recognition model to convert it into text commands, which are then executed by the operating system or application. Existing smart devices typically integrate voice assistants, such as system-level voice assistants in mobile terminals, voice assistants in smart speakers with displays, in-vehicle voice assistants in automotive systems, and voice control functions in smart TVs. These voice assistants can typically perform operations such as making phone calls, sending messages, checking the weather, opening applications, playing music, adjusting volume, and controlling some smart home devices, thus improving the ease of use of smart devices to a certain extent.

[0004] However, existing voice assistants primarily cater to the operating system's built-in functions or applications that have proactively adapted to voice interfaces. For third-party applications that do not provide a dedicated voice control interface beforehand, existing voice assistants can typically only perform external launch operations such as opening the application, making it difficult to perform fine-grained control over visible interactive objects within the third-party application. For example, in short video applications, users typically need to swipe up, down, left, right, click the like button, click the comment button, or click the pause / play area to complete the internal interactions within the application. Even if existing voice assistants can open the short video application, they usually cannot directly perform these internal operations through natural speech.

[0005] To achieve internal application operations, some solutions employ automated scripts, click simulation tools, accessibility service tools, or external hardware controllers to perform actions such as swiping and clicking using preset coordinates or scripts. While these solutions can simulate manual user operations to some extent, they typically require users to manually configure scripts, coordinates, or processes, making it difficult to dynamically assess the current foreground application, active window, control nodes, input method status, and interface risk status. Especially when application versions are updated, interface layouts change, screen resolutions change, landscape / portrait switching occurs, or application pages navigate, the previously fixed coordinates or scripts can easily become ineffective, and may even incorrectly apply click or swipe events to unexpected interfaces.

[0006] Furthermore, existing voice control solutions typically focus on recognizing and executing voice commands, lacking a comprehensive security control process before and after command execution. For ordinary playback or browsing interfaces, voice-activated swiping and clicking pose relatively low risks; however, for sensitive interfaces such as payment confirmation pages, password input pages, transfer pages, system permission authorization pages, account binding pages, amount input pages, and vehicle control pages, directly executing swiping, clicking, text input, or system setting modification operations based on voice recognition results could lead to risks such as unintended payments, accidental transfers, misauthorization, accidental deletion, and accidental modification of system settings. Therefore, a comprehensive voice control system needs to address not only the question of whether control is possible, but also the questions of on which interfaces control is possible, in what manner control is possible, and whether the control results meet expectations.

[0007] Currently, there is typically no independent multi-channel permission judgment mechanism between speech recognition results, interface context, and application state. Voice assistants or automated scripts generally execute operations based on a single speech recognition result or a single preset rule, lacking multi-channel consistency verification. When speech recognition fails, the current interface changes, the application activity window changes, the input method pops up, or the target control is invisible, the system may still generate and inject input events according to the original instructions, resulting in erroneous operations.

[0008] Meanwhile, existing automated control solutions typically lack a time-limited authorization mechanism for the input event itself. Between the completion of voice recognition and the actual injection of the input event, the current interface may have already changed. For example, the user may have just said "like," but the application may have already redirected to a payment confirmation page or permission authorization page before the event injection. If the event injection module still uses coordinates or click parameters generated under the old interface to perform the operation, it will pose a high security risk. Existing technologies generally lack a one-time execution ticket mechanism that binds the interface state fingerprint, allowed event types, allowed coordinate ranges, valid time, and single-use identifier together, making it difficult to prevent the incorrect use of old instructions, old coordinates, or duplicate tickets in the new interface.

[0009] Furthermore, existing voice control solutions lack adaptability to different applications, windows, and versions. The interface layout, control hierarchy, control text, control descriptions, button positions, and page states of third-party applications can change depending on the application version, device size, system font size, display orientation, and running status. Relying solely on fixed coordinates or scripts can easily lead to problems such as incorrect target control recognition, coordinate offset, invalid clicks, insufficient swipe distance, or accidental touches on system edge gestures. Existing technologies typically do not combine the preceding interface state, voice command set, event parameter generation rules, post-state verification conditions, and recipe version number into a manageable application operation recipe. They also lack a post-verification mechanism to check whether the target control state, video identifier, page identifier, input box content, or system settings have changed after execution.

[0010] The shortcomings of existing technology: First, existing voice assistants lack sufficient control over the internal interactions of third-party applications that have not actively adapted to the voice interface, and usually find it difficult to directly control visible interactive objects within the application such as swiping, clicking, text input, and navigation. Second, automation solutions based on fixed coordinates or fixed scripts are less adaptable to changes in application version, window state, resolution, display orientation, and input method state, and are prone to false triggers or operation failures after interface changes. Third, the existing solution lacks multi-channel permission judgment and hierarchical security control that combines voice recognition results, interface context and application status, making it difficult to adopt different execution strategies between ordinary browsing interfaces and sensitive interfaces such as payment, password, and permission authorization. Fourth, the existing solution lacks a closed-loop control mechanism that combines event authorization, time-limited and domain-limited execution, and post-state verification, making it difficult to detect problems such as old instructions being applied to new interfaces, coordinate out-of-bounds errors, duplicate use of tickets, or execution results not meeting expectations in a timely manner. Summary of the Invention: To address the problems existing in the prior art, this invention provides a voice-driven global control system and method for intelligent devices.

[0011] Technical solution This invention provides a voice-driven global control method for a smart device. The smart device includes a device body, a display screen disposed on the device body, a voice input component disposed on or communicatively connected to the device body and used for collecting user voice, a touch input component superimposed or coupled to the display screen or communicatively connected to the device body, a processor, and a memory. The memory stores program instructions executable by the processor. The method is executed by the processor and includes the following steps: S1. The voice input component continuously monitors for wake words or receives wake-up operations triggered by the user. After detecting the wake word or the wake-up operation, the voice recognition engine is activated to convert the user's voice input into text commands. S2. Obtain the interface context of the smart device. The interface context includes at least the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and sensitive interface determination information. S3. Determine the voice intent type based on the text instruction. The voice intent type includes at least one of swipe intent, click intent, text input intent, system call intent, and navigation intent. S4. Generate candidate system-level input event parameters based on the voice intent type and the interface context. When the voice intent type is a swipe intent, with the upper left corner of the display screen as the origin, the width direction as the positive x-axis, and the height direction as the positive y-axis, determine the swipe start point, swipe end point, swipe duration, and swipe trajectory based on the width W and height H of the display screen. Here, W represents the effective display width of the display screen, and H represents the effective display height of the display screen. A controlled perturbation is added to at least one of the swipe start point, the swipe duration, and the swipe trajectory to reduce the risk of operation failure caused by fixed coordinates and fixed trajectories, and to form swipe event parameters that conform to the characteristics of touch input. The candidate system-level input event parameters record the origin of the coordinates, the coordinate unit, the event type, the event duration, and the target object identifier. S5. Before injecting system-level input events, perform permission checks on the text instructions, the interface context, and the candidate system-level input event parameters; wherein, the permission checks include sensitive interface checks, three-channel consistency permission checks, and security permission matrix checks; when the above permission checks pass and system-level input events need to be injected, if the current interface state is consistent with the interface state when the candidate system-level input event parameters are generated, generate and verify a one-time execution ticket bound to the current interface state, allowed event type, allowed coordinate range, and effective time window; S6. When both the permission check and the one-time execution ticket verification pass, inject a system-level input event into the current foreground application through a user-authorized accessibility service interface or an input assistance interface permitted by the operating system, based on the system-level input event parameters that passed the permission check. This enables voice control of visible interactive objects within the current foreground application without requiring the current foreground application to provide a dedicated voice control interface in advance. When either the permission check or the one-time execution ticket verification fails, the injection of the corresponding system-level input event is refused. S7. After the system-level input event is executed, the interface context of the currently active window is retrieved again, and the success of the system-level input event is determined according to the post-validation conditions. If the success is determined, the exception isolation strategy is executed. S8. Output multimodal feedback, wherein the multimodal feedback includes at least one of voice broadcast, operation position highlighting prompt on the display screen, and vibration prompt; S9. During the continuous listening period after a single wake-up, wait for the next text command. If no timeout occurs and no exit command is received, return to step S2. If the timeout occurs or an exit command is received, end the continuous listening.

[0012] Further, in step S2, the accessibility node tree includes the control type, control text, control description, control boundary coordinates, clickable attributes, input attributes, and control hierarchy of each visible control within the currently active window; the input method status includes whether the virtual keyboard is displayed; the sensitive interface determination information is determined by the processor based on at least one of the following in the accessibility node tree: password input control, payment confirmation keyword, transfer keyword, amount input control, permission authorization keyword, account binding keyword, identity verification keyword, vehicle control keyword, and financial application identifier. When the currently active window is identified as a payment confirmation page, password input page, transfer page, system permission authorization page, account binding page, identity verification page, vehicle control page while the vehicle is in motion, or a financial operation page containing amount input controls, the currently active window is marked as a sensitive interface. Under the sensitive interface, only navigation-type security commands such as return, cancel, close, and return to the desktop are allowed to be executed. When a swipe command, click command, or text injection command is detected, the corresponding system-level input event is refused to be generated or injected, and the user is prompted by voice broadcast or flashing of the screen border that the current interface does not support the operation.

[0013] Further, in step S4, when the voice intent type is a swipe intent, the processor determines the swipe start point and swipe end point in the coordinate system as follows: When the text command corresponds to an upward swipe, the starting point of the swipe is determined to be (W / 2, H×0.7), and the ending point of the swipe is determined to be (W / 2, H×0.3). When the text command corresponds to a downward slide, the starting point of the slide is determined to be (W / 2, H×0.3), and the ending point of the slide is (W / 2, H×0.7). When the text command corresponds to a left swipe, the starting point of the swipe is determined to be (W×0.7, H / 2), and the ending point of the swipe is determined to be (W×0.3, H / 2). When the text command corresponds to a right swipe, the starting point of the swipe is determined to be (W×0.3, H / 2), and the ending point of the swipe is determined to be (W×0.7, H / 2). The controlled perturbation includes: adding a position offset of no more than ±5 pixels to the horizontal and vertical coordinates of the sliding start point, adding a duration offset of no more than ±30 milliseconds to the sliding duration, and adding a trajectory curvature offset of no more than ±10 pixels to the sliding trajectory; the processor constructs a Bézier curve path or a sinusoidal perturbation path based on the sliding start point, sliding end point, and trajectory curvature after adding the controlled perturbation, and then injects the corresponding sliding event through a user-authorized accessibility service interface or an input assistance interface allowed by the operating system.

[0014] Further, in step S4, when the voice intent type is a click intent, the processor first searches for control text or control description that matches the text instruction in the accessibility node tree, and when a matching control is found, it generates click event parameters using the boundary coordinate center of the matching control as the click position; when no matching control is found but the current foreground application and the current active window match the preset scene, it calls the preset coordinates corresponding to the preset scene as the click position to generate click event parameters. When the voice intent type is text input intent, the processor determines whether text injection is allowed based on the inputtable controls in the current active window, the input method status, and the sensitive interface determination information. When the target input box is a password input box, verification code input box, payment input box, or amount input box, text injection via voice is prohibited. When the target input box is a regular text input box and the current active window is not marked as a sensitive interface, text injection is allowed after voice verification or touch confirmation, and the boundaries of the target input box and the effective time window are limited by a one-time execution ticket before injection.

[0015] Further, in step S5, the three-channel consistency permission judgment and the security permission matrix judgment include: Establish semantic channels, interface channels, and state channels; The semantic channel generates a first permission result based on the text command, speech recognition confidence level, and speech intent type. The first permission result includes a semantic permission result or a semantic rejection result. When the speech recognition confidence level is lower than a preset confidence threshold, the semantic channel generates a semantic rejection result. The interface channel generates an interface state fingerprint based on the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and the screen orientation. It then generates a second permission result based on the interface state fingerprint and the control text, control description, control boundary coordinates, and control clickable attributes of the target control. The second permission result includes an interface permission result or an interface rejection result. The status channel generates a third permission result based on the current active window's previous status, the allowed action corresponding to the text command, the expected subsequent status, and the preset verification window. The third permission result includes a status permission result or a status rejection result. The processor votes based on the first permission result, the second permission result, and the third permission result: when the voice intent type is a normal swipe intent and the current active window is not marked as a sensitive interface, a two-out-of-three voting rule is adopted; when the voice intent type is a click intent, a text input intent, a system call intent, or the risk level of the current active window is not lower than a preset risk level, a three-out-of-three voting rule is adopted. The processor determines the interface risk level R based on the interface context of the currently active window and the operation level A based on the voice intent type, where R is a discretized interface risk level and A is a discretized operation level. R includes at least a normal browsing interface, a normal application interaction interface, a normal text input interface, a sensitive confirmation interface, and a high-risk control interface. A includes at least navigation security operations, normal swiping operations, normal clicking operations, text input or system setting modification operations, and payment confirmation, permission granting, or data deletion operations. The processor queries a preset security permission matrix M[R, A] to obtain the execution strategy, where M[R, A] is a matrix item in the preset security permission matrix corresponding to the interface risk level R and the operation level A. The execution strategy includes one of the following: direct permission, voice verification, touch confirmation, refusal to execute, and protection lock. When the execution policy is direct permission, the processor is allowed to inject system-level input events; when the execution policy is voice verification, the processor is allowed to inject system-level input events after the user speaks the confirmation command within a preset verification duration; when the execution policy is touch confirmation, the processor is allowed to inject system-level input events after the user completes manual touch confirmation within the confirmation area of ​​the display screen; when the execution policy is denial of execution, the processor is prohibited from injecting system-level input events; when the execution policy is protection lock, voice commands other than navigation safety commands are prohibited from being executed within a preset protection lock duration.

[0016] Further, in step S5, the one-time execution of ticket verification includes: The processor generates an execution permission request based on the candidate system-level input event parameters determined by the three-channel consistency permission judgment and the security permission matrix. The execution permission request includes text instructions, speech recognition confidence, speech intent type, application identifier of the current foreground application, current active window identifier, interface state fingerprint, candidate input event type, candidate input event coordinates or trajectory, candidate input event duration, and current timestamp. A one-time execution ticket is generated based on the execution license request, and a ticket signature is generated on the digest data of the execution license request using a preset key or a key stored in the system's trusted storage; the one-time execution ticket includes a ticket number, an interface status fingerprint, allowed event types, allowed coordinate ranges, a valid time window, a single-use identifier, and a ticket signature; Before injecting system-level input events, the processor verifies the ticket signature, valid time window, interface state fingerprint, allowed event types, allowed coordinate range, and single-use identifier of the one-time execution ticket; When the ticket signature verification fails, the current time exceeds the valid time window, the current interface state fingerprint is inconsistent with the interface state fingerprint in the one-time execution ticket, the type of the event to be injected does not belong to the allowed event type, the coordinates or trajectory to be injected exceed the allowed coordinate range, or the single-use identifier indicates that the one-time execution ticket has been used, the processor refuses to inject system-level input events. When the one-time execution ticket is verified, the processor injects a system-level input event only once according to the allowed event type, allowed coordinate range and valid time window defined by the one-time execution ticket, and updates the single-use identifier to the used state after the injection is completed; The effective time window is 300 milliseconds to 1000 milliseconds, and is calculated from the moment the ticket is generated in one execution; the allowed coordinate range is a rectangular area defined by the boundary coordinates of the target control, or a circular area with the center point of the target control as the center and a radius of 30 to 80 pixels; when the event to be injected is a sliding event, the allowed coordinate range is used to define the sliding start point, sliding end point, and sampling points on the sliding trajectory.

[0017] Furthermore, the processor establishes application operation recipes for different foreground applications and different active windows, and generates, verifies, and validates system-level input events based on the application operation recipes; Each of the application operation recipes includes a recipe number, an applicable application identifier, an applicable active window identifier, an applicable screen orientation, an applicable resolution range, a front-end interface state fingerprint, a voice command set, event parameter generation rules, post-execution verification conditions, and a recipe version number; When the application identifier, current active window identifier, screen orientation, and display resolution of the current foreground application match the target application operation recipe, and the current interface state fingerprint matches the foreground interface state fingerprint in the target application operation recipe, the processor generates candidate system-level input event parameters according to the event parameter generation rules in the target application operation recipe. When the target application operation recipe corresponds to the click intent, the event parameter generation rules include: firstly matching the target control based on the control text or control description in the accessibility node tree, and using the boundary coordinate center of the target control as the click position; when no target control is matched and the current active window belongs to the preset full-screen operation scenario, the click position is generated according to normalized coordinates. When the target application operation recipe corresponds to a sliding intention, the event parameter generation rules include: determining the sliding start point, sliding end point, sliding distance, and sliding trajectory based on the display width W and height H, and ensuring that the sliding distance for upward or downward sliding is not less than 25% and not greater than 60% of the display height H, the sliding distance for left or right sliding is not less than 25% and not greater than 60% of the display width W, and the distance between the sliding trajectory and the edge of the display is not less than 5% of the corresponding screen width W or height H; Within the preset verification window after the processor injects the system-level input event, the processor re-acquires the accessibility node tree and interface state fingerprint of the currently active window, and determines whether the system-level input event was successfully executed based on the post-verification conditions. The post-verification conditions include at least one of the following: change in the state of the target control, change in the text of the target control, change in the attribute of the target control, change in the current video identifier, change in the current page identifier, appearance of the comment panel, change in the content of the input box, and change in the system setting value. When the post-verification condition is not met, the processor marks the current system-level input event as an abnormal execution event and executes an abnormal isolation strategy. The abnormal isolation strategy includes at least one of the following: pausing continuous listening, prohibiting the repeated execution of the same type of voice command within a preset time, reacquiring the current interface context, prompting the user to reconfirm, or switching the current application operation formula to a downgraded mode. The degradation mode includes: prioritizing the use of the accessibility node tree to rematch the target control; using the icon area recognition result in the screenshot to determine the target control when the accessibility node tree is unavailable; and refusing to execute the corresponding click intent and outputting an operation failure message when neither the accessibility node tree nor the screenshot can determine the target control, while marking the current application operation recipe as pending update.

[0018] Furthermore, the voice monitoring, voice recognition, and system-level input event generation processes of the voice input component are executed in parallel with the process of the touch input component receiving user manual touch operations; when the user touches the display screen during the execution of a system-level input event, the touch input component continues to respond to the user's manual touch operation, and the processor does not terminate voice monitoring due to the user's manual touch operation; when the user's manual touch operation causes a change in the interface state fingerprint, the processor invalidates the generated but not injected one-time execution ticket and returns to step S2; The present invention also provides a voice-driven global control system for intelligent devices to implement the above method, including a device body, a display screen, a voice input component, a touch input component, a processor, a memory, and a vibration component; The display screen is located on the front side of the main body of the device and is used to display the active window of the current foreground application and highlight the operation position. The voice input component is disposed on the main body of the device, or disposed on a remote control, an external microphone, or a wearable acquisition device and electrically or communicatively connected to the processor, for the purpose of acquiring user voice. The touch input component is stacked or coupled to the display screen, or disposed on the remote control touch area, touchpad or external pointer input device, and is electrically or communicatively connected to the processor to receive manual touch operations from the user. The vibration component is disposed inside the main body of the device and electrically connected to the processor, and is used to output vibration alerts; The memory stores program instructions that can be executed by the processor. When the processor executes the program instructions, it implements the following functional modules: The voice wake-up and recognition module is used to listen for wake-up words or receive wake-up operations triggered by the user, and convert the user's voice into text commands; The context awareness module is used to obtain the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and sensitive interface judgment information. The instruction mapping module is used to map the voice intent type to candidate system-level input event parameters based on the text instruction and the interface context obtained by the context awareness module. The permission control module is used to perform sensitive interface judgment, three-channel consistency permission judgment, and security permission matrix judgment on the text instructions, the interface context, and the candidate system-level input event parameters, and to perform one-time execution ticket generation and verification when system-level input events need to be injected; The event injection module is used to inject system-level input events through the user-authorized accessibility service interface or the operating system-permitted input assistance interface after the permission control module has granted permission and the ticket verification has been successfully completed in one go. The post-state verification module is used to determine whether the system-level input event was successfully executed based on the re-acquired interface context and the post-verification conditions after the system-level input event is executed. The security control module is used to enter a security mode when the currently active window is a sensitive interface, allowing only navigation-type security commands to be executed and prohibiting the execution of system-level input events corresponding to swipe intents, click intents, and text input intents; A multimodal feedback module is used to output at least one of voice broadcast, operation position highlighting and vibration prompts after a system-level input event is executed.

[0019] Furthermore, the smart device is any one of the following: smartphone, tablet computer, smart TV, projector, set-top box, in-vehicle infotainment system, smart cockpit, smartwatch, smart glasses, head-mounted display device, smart speaker with display screen, or smart home control center; When the smart device is a smartphone or tablet, the display screen is a touch screen, and the voice input component is located at the top, bottom, or bezel of the device body; When the smart device is a smart TV, projector, or set-top box, the voice input component is set on the main body of the device, remote control, or external microphone. The event injection module generates corresponding sliding event parameters, directional key event parameters, or focus movement event parameters according to the display resolution or focus layout. When the intelligent device is an in-vehicle infotainment system or a smart cockpit, the voice input component is located near the center console, ceiling, or steering wheel in the vehicle. The command mapping module is also used to map text commands to system call events corresponding to navigation, music playback, volume adjustment, or air conditioning adjustment. When the vehicle is detected to be in motion and the text command corresponds to a vehicle control or safety-related operation, the safety control module raises the operation level to a high-risk operation level and executes a touch confirmation, rejection, or protection lock strategy.

[0020] Compared with the prior art, the present invention has the following advantages: Beneficial effects : I. This invention combines the speech recognition results with the interface context such as the current foreground application, active window, accessibility node tree, and input method status, and converts speech commands into system-level input events that the current interface can respond to. It does not require third-party applications to provide dedicated speech interfaces in advance, thereby improving the control over visible interactive objects within third-party applications.

[0021] Second, this invention generates event parameters such as click, swipe, and text input based on the width and height of the display screen, the coordinates of the control boundaries, and the application operation formula. It also adds controlled perturbations to the swipe start point, duration, and trajectory, so that the event parameters can be adaptively adjusted according to the resolution, window state, and control position, reducing the risk of operation failure caused by fixed coordinates and fixed trajectories.

[0022] Third, this invention uses sensitive interface judgment, three-channel consistency permission judgment and security permission matrix to perform layered verification of semantic reliability, interface executability and state transition relationship, and outputs direct permission, voice verification, touch confirmation, refusal to execute or protection lock strategy for different interface risk levels and operation levels, thereby improving the security of voice control.

[0023] Fourth, this invention binds the interface state fingerprint, allowed event types, allowed coordinate ranges, valid time windows, single-use identifiers and ticket signatures to a one-time execution ticket, and verifies them before event injection, which can prevent old instructions, old coordinates or duplicate tickets from being used incorrectly in the new interface.

[0024] Fifth, this invention applies operation recipes, post-state verification, and multimodal feedback to reacquire the interface context after event execution and determine whether the target control state, page identifier, input box content, or system setting value meets expectations; when verification fails, exception isolation or degradation mode is executed, thereby forming a closed-loop control of "authorization-execution-verification-feedback". Attached Figure Description

[0025] Figure 1 This is a general structural diagram of the global control system for voice-driven intelligent devices of the present invention; Figure 2 This is a flowchart of the voice command processing and event execution of the present invention; Figure 3 This is a schematic diagram of the desktop voice entry icon and function panel of the present invention; Figure 4 This is a schematic diagram of the voice-controlled short video application of the present invention; Figure 5 This is a schematic diagram of the sliding event parameter generation of the present invention. Detailed Implementation

[0026] The following is combined with Figures 1 to 5 The present invention will be further described below. It is understood that the specific embodiments described are for illustrative purposes only and are not intended to limit the scope of protection. For ease of description, only the parts relevant to the present invention are shown in the accompanying drawings.

[0027] Example 1 : Use voice control to perform actions such as swiping, liking, and continuous operation in short video apps on your smartphone. This embodiment uses a smartphone running a short video application as an example for illustration. The smartphone includes a device body 102, a display screen 101, a voice input component 10, a touch input component 103, a processor 104, a memory 105, and a vibration component 106. The device body 102 has a rectangular plate-like structure. The display screen 101 is disposed on the front surface of the device body 102 and is adapted to the front shape of the device body 102. The touch input component 103 is a transparent touch layer, which is stacked on the light-emitting side of the display screen 101, or integrated with the display screen 101 to form a touch display screen, used to receive manual touch operations such as clicking, long pressing, and swiping by the user's fingers. The voice input component 10 includes at least one microphone, which is disposed on the bottom bezel, top bezel, or side bezel of the device body 102. The microphone is located in the pickup hole on the bottom edge of the device body 102. The pickup hole is a circular or strip-shaped through hole structure. The microphone's pickup surface faces the outside of the device body 102 and is used to collect user voice. The processor 104 and memory 105 are located on the main board inside the device body 102. The processor 104 is electrically connected to the display screen 101, touch input component 103, voice input component 10, and vibration component 106. The vibration component 106 is located inside the device body 102 and adjacent to the lower middle part of the device body 102. It is used to generate short-term vibration feedback after the operation is completed.

[0028] In this embodiment, the smartphone's display screen 101 has a resolution of 1080×2400 pixels, with a screen width W of 1080 pixels and a screen height H of 2400 pixels. The smartphone runs the Android operating system and has accessibility service permissions enabled. The memory 105 stores a voice wake-up and recognition program, a context-aware program, an instruction mapping program, a security control program, an event injection program, a post-state verification program, and a multimodal feedback program. After the processor 104 executes the above programs, it forms a voice processing component 20, a context-aware component 30, an intent mapping and event generation component 40, a security control component 50, an event injection component 60, a feedback and presentation component 70, and a post-state verification component 80.

[0029] The aforementioned voice processing component 20, context awareness component 30, intent mapping and event generation component 40, security control component 50, event injection component 60, feedback and presentation component 70, and post-state verification component 80 respectively implement the voice wake-up and recognition module, context awareness module, instruction mapping module, security control module, event injection module, multimodal feedback module, and post-state verification module. Each component can be implemented by an independent program unit or by the same processor 104 executing program instructions in memory 105 and implementing them as a software functional module.

[0030] S101, the voice processing component 20 is in a low-power monitoring state. The voice input component 10 continuously collects ambient sound, and the processor 104 calls the local wake-up word detection model to detect whether the user has spoken a preset wake-up word. The preset wake-up word can be "Mobile AI". When no wake-up word is detected, only the low-power monitoring program runs, and the full speech recognition engine is not started; after detecting that the user has spoken "Mobile AI", the voice processing component 20 starts the speech recognition engine and outputs a prompt tone through the speaker, indicating that it has entered a continuous monitoring state that can receive voice commands.

[0031] S102. The user says "swipe up" on the full-screen video playback interface of the short video application. The voice input component 10 collects the voice, and the voice recognition engine converts the voice into the text command "swipe up". To reduce misoperation caused by misrecognition, the voice recognition engine also outputs the recognition confidence score when outputting the text command. In this embodiment, when the recognition confidence score is greater than or equal to 0.80, the text command is sent to the intent mapping and event generation component 40; when the recognition confidence score is less than 0.80, the feedback and presentation component 70 provides a voice prompt "I didn't hear you clearly, please say it again", and does not generate a system-level input event.

[0032] S103. The context-aware component 30 obtains the current interface context. Specifically, the context-aware component 30 obtains the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and sensitive interface determination information. In this embodiment, the application identifier of the current foreground application is the short video application identifier, and the current active window is the short video full-screen playback window; the accessibility node tree includes several visible control nodes, and each control node includes at least the control type, control text, control description, control boundary coordinates, whether the control is clickable, whether the control is input-enabled, and the control hierarchy; the input method status is not displayed; and the sensitive interface determination information is a non-sensitive interface.

[0033] S104. Before generating or injecting system-level input events, the security control component 50 performs a sensitive interface determination on the currently active window. Specifically, the security control component 50 searches the accessibility node tree for password input controls, payment confirmation keywords, transfer keywords, amount input controls, and financial application identifiers. In this embodiment, the currently active window does not have a password input control, nor does it contain sensitive keywords such as "confirm payment," "pay immediately," "confirm transfer," "enter payment password," or "amount," nor does it contain a financial application identifier. Therefore, the security control component 50 determines that the currently active window is not a sensitive interface and allows the execution of the system-level input event corresponding to the swipe intention.

[0034] S105, the intent mapping and event generation component 40 determines the voice intent type as a swipe intent and the swipe direction as upward based on the text command "swipe up". The intent mapping and event generation component 40 reads the display width W and height H and determines the basic swipe start point and basic swipe end point according to the preset swipe coordinate rules. For the "swipe up" command, the basic swipe start point P0 is (W / 2, H×0.7) and the basic swipe end point P2 is (W / 2, H×0.3). In this embodiment, W is 1080 pixels and H is 2400 pixels, therefore the basic swipe start point P0 is (540, 1680) and the basic swipe end point P2 is (540, 720).

[0035] S106. To reduce the risk of operational failure caused by fixed coordinates and fixed trajectories, and to ensure that the injected sliding events conform to the characteristics of touch input, the intent mapping and event generation component 40 adds controlled perturbations to the sliding events. Specifically, it generates position offsets Δx and Δy, where Δx and Δy represent the lateral and longitudinal position offsets, respectively, with values ​​ranging from [-5, 5] pixels; it generates a duration offset ΔT, where ΔT represents the sliding duration offset, with values ​​ranging from [-30, 30] milliseconds; and it generates a trajectory curvature offset ΔC, where ΔC represents the trajectory curvature offset used to determine the offset of the Bezier control points, with values ​​ranging from [-10, 10] pixels. In this embodiment, within a preset range, Δx = 3 pixels, Δy = -4 pixels, ΔT = 18 milliseconds, and ΔC = 6 pixels are generated. Therefore, the actual sliding start point P0' is (543, 1676), and the actual sliding end point Pe' is (540, 720). The base sliding duration is 220 milliseconds. After adding the duration offset, the actual sliding duration T is 238 milliseconds.

[0036] S107, the intent mapping and event generation component 40 constructs a Bézier curve path based on the actual sliding start point P0', the actual sliding end point Pe', and the trajectory curvature offset ΔC. Specifically, the first control point C1 can be set as (W / 2 + ΔC, (the ordinate of P0' + the ordinate of Pe') / 2); when using... Figure 5 When using the dual-control-point curve shown, a second control point C2 can be set, so that C1 and C2 are located at different segments of the line connecting the start and end points. In this embodiment, C1 is (546, 1198). A second- or third-order Bézier curve is formed by P0', at least one control point, and Pe', serving as the sliding trajectory for this upward swipe event. Through this curve path, the sliding trajectory is no longer a perfectly straight line, but has a slight curve, more closely resembling the actual sliding process of the user's finger on the screen.

[0037] S108, the event injection component 60 injects the swipe event through a user-authorized accessibility service interface or an input assistance interface permitted by the operating system. Specifically, in the Android system, the event injection component 60 creates a gesture description object and a swipe description object based on the Bézier curve path constructed in step S107. The start time of the swipe description object is 0 milliseconds, and the duration is 238 milliseconds. Then, it calls the user-authorized accessibility service interface to inject the swipe event into the current foreground application. Since this swipe event is a system-level input event, even if the current foreground application has not pre-connected a dedicated voice control interface, this system-level input event can still be processed by the current interface as an input event corresponding to the user's manual swipe.

[0038] S109. After receiving the swipe event, the short video application switches the currently playing video to the next video according to its own application logic. After the event is completed, the post-state verification component 80 re-acquires the accessibility node tree and interface state fingerprint of the currently active window, and determines that the swipe event was successfully executed when it detects a change in the current video identifier or page identifier; the feedback and presentation component 70 then outputs feedback information. Specifically, the feedback and presentation component 70 can announce "swipe up" in voice, and simultaneously highlight the swipe trajectory or the area near the swipe starting point on the display screen 101 for 100 milliseconds, and control the vibration component 106 to vibrate for 50 milliseconds. The highlighted prompt can be a semi-transparent circular light spot or a short linear trajectory, and its display position corresponds to the operation position of this system-level input event, used to let the user confirm that the voice command has been executed.

[0039] S110. After completing the "swipe up" command, the voice processing component 20 does not immediately exit, but waits for the next voice command during the continuous listening period. In this embodiment, the continuous listening period is 10 seconds; if the user continues to speak a new command within 10 seconds, the timer resets to 10 seconds; if no new voice command is detected within 10 seconds, the continuous listening state is exited and the system returns to the low-power listening state; if the user says "exit," "stop listening," or "cancel voice control," the continuous listening state is exited immediately.

[0040] S111. The user continues to say "like" during the continuous listening period. The speech recognition engine converts the speech into the text command "like". The context-aware component 30 re-obtains the application identifier, active window identifier, and accessibility node tree of the current foreground application, and confirms that the current active window is still a non-sensitive interface. The intent mapping and event generation component 40 determines the speech intent type as a click intent based on the text command "like".

[0041] S112. When the voice intent type is a click intent, the intent mapping and event generation component 40 first searches for a target control in the accessibility node tree whose control text or description matches "like". If a target control whose control text or description contains "like" is found, the boundary coordinates of the target control are read, and the center point of the boundary coordinates of the target control is used as the click position. For example, if the boundary coordinates of the target control are [left=920, top=1980, right=1040, bottom=2220], then the click position is (980, 2100). If no "like" target control is found in the accessibility node tree, but the current foreground application identifier and the current active window match the short video full-screen playback scene, then the preset scene coordinates are used as the click position; the preset scene coordinates are determined proportionally according to the width W and height H of the display screen. In this embodiment, the preset coordinates for "like" are (W×0.907, H×0.875), which is approximately (980, 2100).

[0042] S113. The event injection component 60 injects a click event at the click location through a user-authorized accessibility service interface or an input assistance interface permitted by the operating system. The duration of the click event can be from 50 milliseconds to 120 milliseconds; in this embodiment, the click duration is 80 milliseconds. After receiving the click event, the short video application performs a "like" operation. The post-state verification component 80 determines that the click event was successfully executed based on the change in the state or attribute of the like control; the feedback and presentation component 70 generates a circular highlighted prompt at the click location for 100 milliseconds, while simultaneously controlling the vibration component 106 to vibrate for 50 milliseconds and announcing "Likeed" via voice.

[0043] S114. During the aforementioned voice control process, the touch input component 103 remains available at all times. That is, the voice monitoring, voice recognition, intent mapping, and event injection processes of the voice input component 10 are executed in parallel with the process of the touch input component 103 receiving manual touch operations from the user. If the user touches the display screen 101 with their finger during voice control, for example, manually clicking the video screen to pause the video after a swipe event, the touch input component 103 responds normally to the user's manual touch operation. The processor 104 does not block the manual touch operation, nor does it exit the continuous monitoring state due to the manual touch operation. After the user manually pauses the video, they can continue to say "continue playing." The intent mapping and event generation component 40 maps this text command to a click intent and injects a click event in the center area of ​​the video screen, causing the video to resume playback.

[0044] S115. If a user enters a payment confirmation page, password input page, transfer page, or other sensitive interface while continuously monitoring, the security control component 50 will re-determine the sensitive interface before each generation or injection of a system-level input event. For example, if keywords such as "confirm payment," "enter payment password," or "confirm transfer" appear in the accessibility node tree of the currently active window, or if password input controls and amount input controls appear, the security control component 50 will mark the currently active window as a sensitive interface and enter security mode. In security mode, if the user mistakenly utters non-secure commands such as "swipe up," "like," or "enter password," the event injection component 60 will not inject the corresponding swipe event, click event, or text input event. The feedback and presentation component 70 will announce "This is a sensitive interface; this operation is not supported" and cause the screen border to flash for 0.5 seconds. If the user utters "back," "cancel," "close," or "desktop," the security control component 50 will determine it as a navigation-type secure command and allow the event injection component 60 to execute the corresponding system-level input events for returning, canceling, closing, or returning to the desktop.

[0045] Through the above steps, this embodiment can convert natural language commands such as swiping up, liking, swiping left, swiping right, continuing playback, and exiting into responsive swipe events, click events, or navigation events within the application, even when the third-party short video application does not provide a dedicated voice control interface in advance. This can be achieved by utilizing the current foreground application information, the accessibility node tree of the current active window, and the system-level input event injection mechanism. At the same time, through controlled perturbation, Bézier curve paths, safe modes, continuous monitoring, and multimodal feedback, the voice control process not only conforms to the characteristics of touch input but also avoids accidental execution of dangerous operations on sensitive interfaces, and allows users to intervene manually at any time during the voice control process.

[0046] Example 2: Three-channel licensing, one-time execution ticket and post-verification process This embodiment, based on Embodiment 1, further explains the permission judgment before voice commands are converted into system-level input events, the ticket verification during event injection, and the status verification process after event injection. This embodiment uses a smartphone running a short video application as an example, but the same data structure, judgment rules, and verification process can also be applied to devices such as tablets, smart TVs, in-vehicle infotainment systems, smart cockpits, smart glasses, or smart home control centers.

[0047] In this embodiment, the smartphone's display screen 101 has a resolution of 1080×2400 pixels, a screen width W of 1080 pixels, a screen height H of 2400 pixels, and runs on the Android operating system. Accessibility service permissions are enabled. The voice recognition confidence threshold is set to 0.80; the effective time for executing ticket 202 in one go is set to 800 milliseconds; the post-state verification window is set to 1000 milliseconds; the continuous listening timeout is set to 10 seconds; the protection lock duration is set to 3 seconds; and the touch confirmation duration of the touch confirmation area is set to no less than 80 milliseconds.

[0048] Before executing voice control, the processor creates application operation recipes for the target application through the recipe management module. Each application operation recipe is stored in the application operation recipe library 201 in the form of a data table and is called by the processor 104 when generating candidate input event parameters. In this embodiment, a first application operation recipe is created for the short video full-screen playback interface. The recipe number of the first application operation recipe is P-DY-VIDEO-001, the applicable application identifier is the short video application identifier, the applicable active window identifier is the short video full-screen playback window, the applicable screen orientation is portrait, the applicable resolution range is a width of 720 pixels to 1440 pixels and a height of 1280 pixels to 3200 pixels, and the recipe version number is V1.0.

[0049] The first application operation formula includes the front-end interface state fingerprint generation rule, voice command set, event parameter generation rule, and post-execution verification conditions. The front-end interface state fingerprint 203 generation rule is as follows: F=SHA-256(AppID|ActivityID|NodeTextSet|NodeBoundsSet|InputMethodState|Orientation|ScreenSize); In this dataset, AppID is the application identifier of the current foreground application, ActivityID is the identifier of the current active window, NodeTextSet is the set of visible control text in the current active window, NodeBoundsSet is the set of boundary coordinates of visible controls in the current active window, InputMethodState is the input method state, Orientation is the screen orientation, and ScreenSize is the display resolution. The fields are concatenated using a fixed separator and then input into the SHA-256 algorithm to obtain the interface state fingerprint F. To avoid fingerprint instability caused by changes in control order, NodeTextSet and NodeBoundsSet are sorted in ascending order of ordinate and abscissa of control boundary coordinates before calculation.

[0050] The voice command set in the first application operation formula includes: the swipe-up command set is {"swipe up", "next", "next video", "switch to next video"}; the swipe-down command set is {"swipe down", "previous", "previous video", "return to previous video"}; the like command set is {"like", "give a like", "like"}; the comment entry command set is {"comment", "open comments", "view comments"}; and the playback control command set is {"pause", "continue playing", "play"}.

[0051] The event parameter generation rules in the first application operation recipe include swipe rules and click rules. For an upward swipe command, the swipe start point is (W / 2, H×0.7) and the swipe end point is (W / 2, H×0.3); for a downward swipe command, the swipe start point is (W / 2, H×0.3) and the swipe end point is (W / 2, H×0.7); for a left swipe command, the swipe start point is (W×0.7, H / 2) and the swipe end point is (W×0.3, H / 2); for a right swipe command, the swipe start point is (W×0.3, H / 2) and the swipe end point is (W×0.7, H / 2). The swipe distance for upward or downward swipes must not be less than 25% and not more than 60% of the display height H; the swipe distance for left or right swipes must not be less than 25% and not more than 60% of the display width W; the minimum distance between the swipe trajectory and the edge of the display screen must not be less than 5% of the corresponding screen width W or height H. In this embodiment, the starting point of the upward slide is (540, 1680), the ending point is (540, 720), and the sliding distance is 960 pixels, which accounts for 40% of the display height H, thus satisfying the sliding distance constraint. The horizontal coordinate of the sliding trajectory is 540 pixels, and the distance from both the left and right edges is greater than 54 pixels, thus satisfying the edge safety constraint.

[0052] For the "like" command, the click rules are as follows: First, search the accessibility node tree for target controls whose text or description contains "like" or "favorite". If a target control is found, the center of the target control's boundary coordinates is used as the click location. If no target control is found, but the current foreground application identifier, active window identifier, and foreground interface state fingerprint all match the first application's operation recipe, then normalized coordinates (0.907W, 0.875H) are used as the like click location. On a 1080×2400 pixel display, the normalized coordinates correspond to a like click location of (980, 2100).

[0053] The post-execution verification conditions in the first application operation formula include: for swipe-up or swipe-down events, at least one of the following: the current video identifier changes, the video title text changes, the playback progress node is reset, or the current page state fingerprint changes as expected; for like events, at least one of the following: the selected attribute of the like control changes, the like control icon state changes, the like count text changes, or the target control state fingerprint changes; for comment entry click events, at least one of the following: the comment panel node appears, the comment input box node appears, or the activity window identifier changes to the comment panel window.

[0054] After the user opens the short video app and enters the full-screen video playback interface, they say the wake word "Mobile AI". The voice processing component 20 detects the wake word and enters continuous listening mode. The user then says "Like". The voice processing component 20 converts the user's speech into the text command "Like", and outputs a recognition confidence score of 0.93. Since 0.93 is greater than the preset confidence threshold of 0.80, the processor 104 sends the text command "Like" to the three-channel permission judgment 205.

[0055] The context-aware component 30 obtains the current interface context. In this embodiment, the current foreground application identifier is the short video application identifier, the current active window identifier is the short video full-screen playback window, the input method state is not popped up, the screen orientation is portrait, and the display resolution is 1080×2400 pixels. The context-aware component 30 also obtains the accessibility node tree of the current active window. This accessibility node tree includes several visible control nodes. Each control node includes control type, control text, control description, control boundary coordinates, control clickable attributes, control input attributes, and control hierarchy relationship.

[0056] The recipe management module matches the first application operation recipe P-DY-VIDEO-001 based on the current foreground application identifier, the current active window identifier, screen orientation, and display resolution. Upon successful matching, the recipe management module generates candidate input event parameters according to the click rules in the first application operation recipe. If a target control with the description "like" exists in the accessibility node tree, and the boundary coordinates of this target control are [left=920, top=1980, right=1040, bottom=2220], then the candidate input event type is a click event, the candidate click coordinates are the center point of these boundary coordinates (980, 2100), and the candidate click duration is 80 milliseconds.

[0057] The three-channel permission determination 205 establishes a semantic channel 206, an interface channel 207, and a status channel 208, and outputs a first permission result, a second permission result, and a third permission result, respectively. The semantic channel 206 generates the first permission result based on the text command, speech recognition confidence level, and speech intent type. In this embodiment, the text command is "like," the speech recognition confidence level is 0.93, the speech intent type is click intent, and the target object name is "like control." Since the recognition confidence level of 0.93 is higher than the preset confidence threshold of 0.80, and the text command "like" belongs to the like command set in the first application operation recipe, the semantic channel 206 outputs the semantic permission result.

[0058] Interface channel 207 generates a current interface state fingerprint F1 based on the current foreground application identifier, the current active window identifier, the accessibility node tree, the input method status, the screen orientation, and the display resolution. Interface channel 207 determines whether the current interface state fingerprint F1 matches the foreground interface state fingerprint of the first application operation formula, and determines whether the candidate click coordinates are within the boundary coordinates of the target control. In this embodiment, the candidate click coordinates (980, 2100) are within the rectangular area defined by [left=920, top=1980, right=1040, bottom=2220], and the target control has a clickable attribute; therefore, interface channel 207 outputs the interface permission result.

[0059] The state channel 208 generates a third permission result based on the current active window's previous state, allowed actions, expected subsequent state, and a preset verification window. In this embodiment, the current active window's previous state is a full-screen short video playback state. Allowed actions include swiping up, swiping down, liking, clicking the comment entry, and playback control. The action corresponding to the text command "like" is an allowed action. The expected subsequent state is at least one of the following: a change in the state of the like control, a change in the state of the like icon, or a change in the text of the like count. Therefore, the state channel 208 outputs the state permission result.

[0060] Since the voice intent type is a click intent, processor 104 adopts a three-out-of-three voting rule. Semantic channel 206, interface channel 207, and state channel 208 all output permission results, and the three-channel consistency permission judgment passes. If any of the three channels outputs a rejection result, this click intent will not enter the subsequent event injection process, and the feedback and presentation component 70 will provide a voice prompt: "The current state is not suitable for performing this operation."

[0061] After the three-channel consistency permission judgment is passed, the security control component 50 continues to execute the security permission matrix 204 judgment. The security control component 50 determines the interface risk level R based on the interface context of the current active window and determines the operation level A based on the voice intent type. In this embodiment, the interface risk level R includes R0, R1, R2, R3, and R4. Among them, R0 represents a normal browsing interface; R1 represents a normal application interaction interface; R2 represents an interface containing ordinary text input boxes; R3 represents a sensitive interface containing keywords such as payment, transfer, password, identity verification, and account binding; R4 represents a system permission setting page, payment confirmation page, financial transfer confirmation page, vehicle control interface in vehicle driving state, or other high-risk interfaces. The operation level A includes A0, A1, A2, A3, and A4. Among them, A0 represents navigation-type safe operations such as returning, canceling, closing, or returning to the desktop; A1 represents ordinary swiping operations; A2 represents ordinary clicking operations; A3 represents text input, long press, or system setting modification operations; A4 represents payment confirmation, password input, transfer confirmation, permission granting, data deletion, or factory reset operations.

[0062] In this embodiment, the current active window is a full-screen short video playback window, which does not contain keywords related to payment, transfer, password, amount input, or permission granting, nor does it contain a password input box or an amount input box. Therefore, the interface risk level R is R1. The text command "like" corresponds to a normal click operation, so the operation level A is A2. The security control component 50 queries the security permission matrix M[R, A] and finds that the execution policy corresponding to M[R1, A2] is direct permission. Therefore, the security permission matrix 204 passes the judgment, allowing entry into the one-time execution ticket 202 generation process.

[0063] As a further example of the security permission matrix, when the interface risk level R0 corresponds to a normal browsing page and the operation level A1 corresponds to a normal swipe operation, matrix item M[R0, A1] is directly allowed; when the interface risk level R3 corresponds to a payment confirmation page and the operation level A2 corresponds to a click operation, matrix item M[R3, A2] is touch confirmation or denial of execution; when the interface risk level R4 corresponds to a financial transfer confirmation page or a system permission granting page and the operation level A4 corresponds to payment confirmation, transfer confirmation, or permission granting operation, matrix item M[R4, A4] is denial of execution or protection lock.

[0064] After the security permission matrix 204 passes the judgment, the ticket management module generates an execution permission request based on the candidate input event parameters. The execution permission request includes the text command "like", the voice recognition confidence score of 0.93, the voice intent type click intent, the current foreground application identifier, the current active window identifier, the interface state fingerprint F1, the candidate input event type click event, the candidate input event coordinates (980, 2100), the candidate input event duration of 80 milliseconds, and the current timestamp T1.

[0065] The ticket management module generates a one-time execution ticket (Ticket-001) based on the execution permission request and generates a ticket signature using a preset key or a key stored in the system's trusted database, based on the digest data of the execution permission request. The one-time execution ticket (Ticket-001) includes a ticket number (TicketID-001), an interface state fingerprint (F1), an allowed event type (click event), an allowed coordinate range, a validity period of 800 milliseconds, a single-use identifier (Unused), and a ticket signature. The allowed coordinate range is a rectangular area defined by the boundary coordinates of the target control [left=920, top=1980, right=1040, bottom=2220], or a circular area with a radius of 60 pixels centered at the target control's center point (980, 2100). The ticket signature can be generated using the HMAC-SHA-256 algorithm. The signature input includes the ticket number, interface state fingerprint, allowed event type, allowed coordinate range, validity period, single-use identifier, and timestamp. The signature key is stored in the system's secure storage area.

[0066] Before injecting the click event, the event injection component 60 verifies the one-time execution ticket Ticket-001. The verification includes: whether the ticket signature is correct; whether the difference between the current time and the timestamp T1 is less than or equal to 800 milliseconds; whether the current interface state fingerprint is still F1; whether the type of the event to be injected is a click event; whether the coordinates to be injected (980, 2100) are within the allowed coordinate range; and whether the single-use identifier is Unused.

[0067] If all the above verifications pass, the event injection component 60 injects a click event with a duration of 80 milliseconds at coordinates (980, 2100) through a user-authorized accessibility service interface or an input assistance interface allowed by the operating system. After injection, it updates the single-use identifier of the ticket Ticket-001 to Used to prevent the ticket from being reused. If any verification condition is not met, the event injection component 60 refuses to inject the click event. For example, if the application interface redirects to the payment confirmation page within 800 milliseconds after the user says "like", the current interface state fingerprint changes from F1 to F2. The event injection component 60 refuses to inject the click event because the current interface state fingerprint is inconsistent with the interface state fingerprint in the ticket, and the feedback and presentation component 70 provides a voice prompt: "The interface has changed, the operation has been cancelled."

[0068] After the click event is injected, the post-state validation component 80 re-acquires the accessibility node tree and interface state fingerprint of the currently active window within a preset validation time window of 1000 milliseconds. The post-state validation component 80 determines whether the click event was successfully executed based on the post-validation conditions in the first application operation recipe. For a like event, the post-state validation component 80 detects at least one of the following changes: the selected attribute of the target like control changes from false to true; the icon state of the target like control changes from unselected to selected; the like count text changes; or the target control state fingerprint changes. If any of the above changes occur within 1000 milliseconds, the post-state validation component 80 determines that the like event was successfully executed and records the execution result as Success. The feedback and presentation component 70 displays a circular highlighted prompt at the click location for 100 milliseconds, controls the vibration component 106 to vibrate for 50 milliseconds, and announces "Likeed" via voice.

[0069] If no post-validation condition is met within 1000 milliseconds, the post-validation component 80 determines that the click event execution is abnormal and marks this system-level input event as an abnormal execution event. The processor 104 then executes an exception isolation strategy. The exception isolation strategy includes at least one of the following: pausing continuous listening for 2 seconds, prohibiting repeated execution of the "like" command within 5 seconds, reacquiring the current interface context, prompting the user to reconfirm, or switching the first application operation recipe to a downgraded mode. In downgraded mode, the processor 104 first rereads the accessibility node tree and matches the target control; if the accessibility node tree is unavailable, it acquires a screenshot of the screen and identifies the icon area in the right-side interactive button area; if neither the accessibility node tree nor the screenshot can determine the target control, it refuses to execute the like intention and provides a voice prompt: "The current interface cannot confirm the like location."

[0070] In another scenario, a user is using a payment application and has entered a transfer confirmation page. The context-aware component 30 detects that the accessibility node tree of the currently active window contains keywords such as "confirm transfer," "amount," and "enter payment password," and that there are amount or password input controls present. The security control component 50 determines the current interface risk level to R3 or R4. If the user mistakenly says "like" or "swipe up," the semantic channel 206 may still recognize the click or swipe intent, but the interface channel 207, based on the current application identifier, active window identifier, and interface state fingerprint, determines that the current interface is not the short video full-screen playback window applicable to the first application operation formula, and therefore outputs an interface rejection result; the state channel 208 determines that the current preceding state does not match the allowed action table of the short video application, and outputs a state rejection result. For the click intent, the three-way vote fails, and the processor 104 will not generate a system-level click event.

[0071] Even if a normal swipe intent satisfies a two-out-of-three vote, the security permission matrix 204 will still output a rejection or protection lock based on the interface risk level R3 or R4 and the operation level A1, A2 or A3. The event injection component 60 will not inject swipe events, click events, or text input events. The feedback and presentation component 70 will announce "This is a sensitive interface; this operation is not supported" and cause the display border to flash for 0.5 seconds, while simultaneously invalidating any generated but not injected one-time execution tickets. If the user says "back" or "cancel," the security control component 50 will recognize this instruction as a navigation-type security operation A0 and allow the event injection component 60 to execute the return or cancellation operation so that the user can exit the sensitive interface.

[0072] In this embodiment, before voice commands are converted into system-level input events, they must sequentially undergo application operation recipe matching, three-channel consistency authorization, security authorization matrix judgment, and one-time execution ticket verification. After the system-level input event is injected, post-execution state verification is also required. Therefore, the system does not simply rely on voice recognition results to directly click or swipe the screen, but forms a closed-loop control process involving pre-state confirmation, multi-channel voting, risk matrix authorization, time-limited ticket authorization, event injection, post-state verification, and anomaly isolation. This closed-loop control process avoids voice misrecognition directly triggering erroneous clicks or swipes, prevents old commands from continuing to act on new interfaces after application interface transitions, prevents dangerous operations from being performed on sensitive interfaces, prevents preset coordinates from becoming invalid due to application version changes, control position changes, or node tree changes, and promptly isolates and downgrades anomalies when execution results do not meet expectations.

[0073] Example 3: Optional settings and cross-device deployment of desktop voice control entry point This embodiment further explains the optional settings for the desktop voice control entry point and its cross-device deployment method. To facilitate users in activating the global voice control function, the smart device can display a voice control entry point icon 90 on the desktop. The voice control entry point icon 90 can be a floating icon, a fixed desktop icon, or an entry point icon located in the bottom shortcut bar. After the user double-clicks, long-presses, or speaks a preset wake-up word on the voice control entry point icon 90, the processor 104 starts the voice wake-up and recognition module and enters the aforementioned voice recognition, context awareness, permission judgment, ticket verification, and event injection processes. This entry point icon is only used as a function activation method and is not a necessary condition for the generation, authorization, or injection of system-level input events.

[0074] When a user drags an application icon or system function icon from the semi-transparent function panel 91 to a blank area of ​​the desktop, the processor 104 creates a corresponding desktop shortcut icon in the blank area and records the mapping relationship between the shortcut icon and the application identifier, system function identifier, or voice command set. When the user drags the desktop shortcut icon to the voice control entry icon 90, the processor 104 removes the desktop shortcut icon from the desktop and stores it in the semi-transparent function panel 91. Thus, the voice control entry icon 90 is not only used to activate the global voice control process, but also to centrally manage frequently used applications and system function entries.

[0075] When the smart device is a smart TV, projector, or set-top box, the display screen can be a TV screen, a projected image, or an external display screen. The voice input component 10 can be located on the main body of the device, a remote control, or an external microphone. The touch input component 103 can be the touch area of ​​a remote control, a touchpad, or an external pointer input device. For devices without a touch display screen, the intent mapping and event generation component 40 converts voice intents such as swiping up, swiping down, swiping left, or swiping right into directional key events, focus movement events, confirmation events, or system-level input events equivalent to swipe gestures. The event injection component 60 generates corresponding event parameters based on the display screen resolution, current focus position, focus list order, and target control boundary information. The post-state verification component 80 determines the execution result based on focus changes, page label changes, or system setting value changes.

[0076] When the smart device is a smartwatch, smart glasses, head-mounted display, smart speaker with a display screen, or smart home control hub, the display screen can be a small-sized touchscreen, a near-eye display, a vehicle central control screen, a speaker display area with a screen, or a home central control screen; the voice input component 10 can be set on the main body of the device, wearable frame, external pickup device, or paired remote control; the touch input component 103 can be a touchscreen, touchpad, gesture input component, or external pointer input device. The processor 104 still executes according to the process of voice recognition, context awareness, intent mapping, permission judgment, one-time execution ticket verification, event injection, post-state verification, and multimodal feedback, so that the same global control architecture can be reused in different device forms.

[0077] The above description is merely a preferred embodiment of the present invention and does not limit the scope of protection of the present invention. Any equivalent structural transformations, equivalent process substitutions, equivalent module divisions made using the content of this specification and drawings, or direct or indirect applications in other related technical fields under the concept of the present invention, should be included within the scope of protection of the present invention.

Claims

1. A voice-driven global control method for intelligent devices, characterized in that, The smart device includes a device body, a display screen disposed on the device body, a voice input component disposed on or communicatively connected to the device body and used for collecting user voice, a touch input component superimposed or coupled to the display screen or communicatively connected to the device body, a processor, and a memory. The memory stores program instructions executable by the processor. The method is executed by the processor and includes the following steps: S1. The voice input component continuously monitors for wake words or receives wake-up operations triggered by the user. After detecting the wake word or the wake-up operation, the voice recognition engine is activated to convert the user's voice input into text commands. S2. Obtain the interface context of the smart device. The interface context includes at least the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and sensitive interface determination information. S3. Determine the voice intent type based on the text instruction. The voice intent type includes at least one of swipe intent, click intent, text input intent, system call intent, and navigation intent. S4. Generate candidate system-level input event parameters based on the voice intent type and the interface context. When the voice intent type is a swipe intent, with the upper left corner of the display screen as the origin, the width direction as the positive x-axis, and the height direction as the positive y-axis, determine the swipe start point, swipe end point, swipe duration, and swipe trajectory based on the width W and height H of the display screen. Here, W represents the effective display width of the display screen, and H represents the effective display height of the display screen. A controlled perturbation is added to at least one of the swipe start point, the swipe duration, and the swipe trajectory to reduce the risk of operation failure caused by fixed coordinates and fixed trajectories, and to form swipe event parameters that conform to the characteristics of touch input. The candidate system-level input event parameters record the origin of the coordinates, the coordinate unit, the event type, the event duration, and the target object identifier. S5. Before injecting system-level input events, perform permission checks on the text instructions, the interface context, and the candidate system-level input event parameters; wherein, the permission checks include sensitive interface checks, three-channel consistency permission checks, and security permission matrix checks; when the above permission checks pass and system-level input events need to be injected, if the current interface state is consistent with the interface state when the candidate system-level input event parameters are generated, generate and verify a one-time execution ticket bound to the current interface state, allowed event type, allowed coordinate range, and effective time window; S6. When both the permission check and the one-time execution ticket verification pass, inject a system-level input event into the current foreground application through a user-authorized accessibility service interface or an input assistance interface permitted by the operating system, based on the system-level input event parameters that passed the permission check. This enables voice control of visible interactive objects within the current foreground application without requiring the current foreground application to provide a dedicated voice control interface in advance. When either the permission check or the one-time execution ticket verification fails, the injection of the corresponding system-level input event is refused. S7. After the system-level input event is executed, the interface context of the currently active window is retrieved again, and the success of the system-level input event is determined according to the post-validation conditions. If the success is determined, the exception isolation strategy is executed. S8. Output multimodal feedback, wherein the multimodal feedback includes at least one of voice broadcast, operation position highlighting on the display screen, and vibration prompt; S9. During the continuous listening period after a single wake-up, wait for the next text command. If no timeout occurs and no exit command is received, return to step S2. If the timeout occurs or an exit command is received, end the continuous listening.

2. The voice-driven global control method for intelligent devices according to claim 1, characterized in that, In step S2, the accessibility node tree includes the control type, control text, control description, control boundary coordinates, clickable attributes, input attributes, and control hierarchy of each visible control within the currently active window; the input method status includes whether the virtual keyboard is displayed; the sensitive interface determination information is determined by the processor based on at least one of the following in the accessibility node tree: password input control, payment confirmation keyword, transfer keyword, amount input control, permission authorization keyword, account binding keyword, identity verification keyword, vehicle control keyword, and financial application identifier. When the currently active window is identified as a payment confirmation page, password input page, transfer page, system permission authorization page, account binding page, identity verification page, vehicle control page while the vehicle is in motion, or a financial operation page containing amount input controls, the currently active window is marked as a sensitive interface. Under the sensitive interface, only navigation-type security commands such as return, cancel, close, and return to the desktop are allowed to be executed. When a swipe command, click command, or text injection command is detected, the corresponding system-level input event is refused to be generated or injected, and the user is prompted by voice broadcast or flashing of the screen border that the current interface does not support the operation.

3. The voice-driven global control method for intelligent devices according to claim 1, characterized in that, In step S4, when the voice intent type is a swipe intent, the processor determines the swipe start point and swipe end point in the coordinate system as follows: When the text command corresponds to an upward swipe, the starting point of the swipe is determined to be (W / 2, H×0.7), and the ending point of the swipe is (W / 2, H×0.3). When the text command corresponds to a downward slide, the starting point of the slide is determined to be (W / 2, H×0.3), and the ending point of the slide is (W / 2, H×0.7). When the text command corresponds to a left swipe, the starting point of the swipe is determined to be (W×0.7, H / 2), and the ending point of the swipe is determined to be (W×0.3, H / 2). When the text command corresponds to a right swipe, the starting point of the swipe is determined to be (W×0.3, H / 2), and the ending point of the swipe is determined to be (W×0.7, H / 2). The controlled perturbation includes: adding a position offset of no more than ±5 pixels to the horizontal and vertical coordinates of the sliding start point, adding a duration offset of no more than ±30 milliseconds to the sliding duration, and adding a trajectory curvature offset of no more than ±10 pixels to the sliding trajectory; the processor constructs a Bézier curve path or a sinusoidal perturbation path based on the sliding start point, sliding end point, and trajectory curvature after adding the controlled perturbation, and then injects the corresponding sliding event through a user-authorized accessibility service interface or an input assistance interface allowed by the operating system.

4. The voice-driven global control method for intelligent devices according to claim 1, characterized in that, In step S4, when the voice intent type is a click intent, the processor first searches for control text or control description that matches the text instruction in the accessibility node tree, and when a matching control is found, it generates click event parameters using the boundary coordinate center of the matching control as the click position; when no matching control is found but the current foreground application and the current active window match the preset scene, it calls the preset coordinates corresponding to the preset scene as the click position to generate click event parameters. When the voice intent type is text input intent, the processor determines whether text injection is allowed based on the input controls in the current active window, the input method status, and the sensitive interface determination information; when the target input box is a password input box, verification code input box, payment input box, or amount input box, text injection via voice is prohibited. When the target input box is a regular text input box and the currently active window is not marked as a sensitive interface, text injection is allowed after voice review or touch confirmation, and the boundaries of the target input box and the effective time window are limited by a one-time execution ticket before injection.

5. The voice-driven global control method for intelligent devices according to claim 1, characterized in that, In step S5, the three-channel consistency permission judgment and the security permission matrix judgment include: Establish semantic channels, interface channels, and state channels; The semantic channel generates a first permission result based on the text command, speech recognition confidence level, and speech intent type. The first permission result includes a semantic permission result or a semantic rejection result. When the speech recognition confidence level is lower than a preset confidence threshold, the semantic channel generates a semantic rejection result. The interface channel generates an interface state fingerprint based on the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and the screen orientation. It then generates a second permission result based on the interface state fingerprint and the control text, control description, control boundary coordinates, and control clickable attributes of the target control. The second permission result includes an interface permission result or an interface rejection result. The status channel generates a third permission result based on the current active window's previous status, the allowed action corresponding to the text command, the expected subsequent status, and the preset verification window. The third permission result includes a status permission result or a status rejection result. The processor votes based on the first permission result, the second permission result, and the third permission result: when the voice intent type is a normal swipe intent and the current active window is not marked as a sensitive interface, a two-out-of-three voting rule is adopted; when the voice intent type is a click intent, a text input intent, a system call intent, or the risk level of the current active window is not lower than a preset risk level, a three-out-of-three voting rule is adopted. The processor determines the interface risk level R based on the interface context of the currently active window and the operation level A based on the voice intent type, where R is a discretized interface risk level and A is a discretized operation level. R includes at least a normal browsing interface, a normal application interaction interface, a normal text input interface, a sensitive confirmation interface, and a high-risk control interface. A includes at least navigation security operations, normal swiping operations, normal clicking operations, text input or system setting modification operations, and payment confirmation, permission granting, or data deletion operations. The processor queries a preset security permission matrix M[R, A] to obtain the execution strategy, where M[R, A] is a matrix item in the preset security permission matrix corresponding to the interface risk level R and the operation level A. The execution strategy includes one of the following: direct permission, voice verification, touch confirmation, refusal to execute, and protection lock. When the execution policy is direct permission, the processor is allowed to inject system-level input events; when the execution policy is voice verification, the processor is allowed to inject system-level input events after the user speaks the confirmation command within a preset verification duration; when the execution policy is touch confirmation, the processor is allowed to inject system-level input events after the user completes manual touch confirmation within the confirmation area of ​​the display screen; when the execution policy is denial of execution, the processor is prohibited from injecting system-level input events; when the execution policy is protection lock, voice commands other than navigation safety commands are prohibited from being executed within a preset protection lock duration.

6. The voice-driven global control method for intelligent devices according to claim 5, characterized in that, In step S5, the one-time execution of ticket verification includes: The processor generates an execution permission request based on the candidate system-level input event parameters determined by the three-channel consistency permission judgment and the security permission matrix. The execution permission request includes text instructions, speech recognition confidence, speech intent type, application identifier of the current foreground application, current active window identifier, interface state fingerprint, candidate input event type, candidate input event coordinates or trajectory, candidate input event duration, and current timestamp. A one-time execution ticket is generated based on the execution license request, and a ticket signature is generated on the digest data of the execution license request using a preset key or a key stored in the system's trusted storage; the one-time execution ticket includes a ticket number, an interface status fingerprint, allowed event types, allowed coordinate ranges, a valid time window, a single-use identifier, and a ticket signature; Before injecting system-level input events, the processor verifies the ticket signature, valid time window, interface state fingerprint, allowed event types, allowed coordinate range, and single-use identifier of the one-time execution ticket; When the ticket signature verification fails, the current time exceeds the valid time window, the current interface state fingerprint is inconsistent with the interface state fingerprint in the one-time execution ticket, the type of the event to be injected does not belong to the allowed event type, the coordinates or trajectory to be injected exceed the allowed coordinate range, or the single-use identifier indicates that the one-time execution ticket has been used, the processor refuses to inject system-level input events. When the one-time execution ticket is verified, the processor injects a system-level input event only once according to the allowed event type, allowed coordinate range and valid time window defined by the one-time execution ticket, and updates the single-use identifier to the used state after the injection is completed; The effective time window is 300 milliseconds to 1000 milliseconds, and is calculated from the moment the ticket is generated in one execution; the allowed coordinate range is a rectangular area defined by the boundary coordinates of the target control, or a circular area with the center point of the target control as the center and a radius of 30 to 80 pixels; when the event to be injected is a sliding event, the allowed coordinate range is used to define the sliding start point, sliding end point, and sampling points on the sliding trajectory.

7. The voice-driven global control method for intelligent devices according to claim 6, characterized in that, The processor creates application operation recipes for different foreground applications and different active windows, and generates, verifies, and validates system-level input events based on the application operation recipes. Each of the application operation recipes includes a recipe number, an applicable application identifier, an applicable active window identifier, an applicable screen orientation, an applicable resolution range, a front-end interface state fingerprint, a voice command set, event parameter generation rules, post-execution verification conditions, and a recipe version number; When the application identifier, current active window identifier, screen orientation, and display resolution of the current foreground application match the target application operation recipe, and the current interface state fingerprint matches the foreground interface state fingerprint in the target application operation recipe, the processor generates candidate system-level input event parameters according to the event parameter generation rules in the target application operation recipe. When the target application operation recipe corresponds to the click intent, the event parameter generation rules include: firstly matching the target control based on the control text or control description in the accessibility node tree, and using the boundary coordinate center of the target control as the click position; when no target control is matched and the current active window belongs to the preset full-screen operation scenario, the click position is generated according to normalized coordinates. When the target application operation recipe corresponds to a sliding intention, the event parameter generation rules include: determining the sliding start point, sliding end point, sliding distance, and sliding trajectory based on the display width W and height H, and ensuring that the sliding distance for upward or downward sliding is not less than 25% and not greater than 60% of the display height H, the sliding distance for left or right sliding is not less than 25% and not greater than 60% of the display width W, and the distance between the sliding trajectory and the edge of the display is not less than 5% of the corresponding screen width W or height H; Within the preset verification window after the processor injects the system-level input event, the processor re-acquires the accessibility node tree and interface state fingerprint of the currently active window, and determines whether the system-level input event was successfully executed based on the post-verification conditions. The post-verification conditions include at least one of the following: change in the state of the target control, change in the text of the target control, change in the attribute of the target control, change in the current video identifier, change in the current page identifier, appearance of the comment panel, change in the content of the input box, and change in the system setting value. When the post-verification condition is not met, the processor marks the current system-level input event as an abnormal execution event and executes an abnormal isolation strategy. The abnormal isolation strategy includes at least one of the following: pausing continuous listening, prohibiting the repeated execution of the same type of voice command within a preset time, reacquiring the current interface context, prompting the user to reconfirm, or switching the current application operation formula to a downgraded mode. The degradation mode includes: prioritizing the use of the accessibility node tree to rematch the target control; using the icon area recognition result in the screenshot to determine the target control when the accessibility node tree is unavailable; and refusing to execute the corresponding click intent and outputting an operation failure message when neither the accessibility node tree nor the screenshot can determine the target control, while marking the current application operation recipe as pending update.

8. The voice-driven global control method for intelligent devices according to claim 1, characterized in that, The voice input component's voice monitoring, voice recognition, and system-level input event generation processes are executed in parallel with the touch input component's process of receiving user manual touch operations. When the user touches the display screen during the system-level input event execution process, the touch input component continues to respond to the user's manual touch operation, and the processor does not terminate voice monitoring due to the user's manual touch operation. When the user's manual touch operation causes a change in the interface state fingerprint, the processor invalidates the generated but not injected one-time execution ticket and returns to step S2.

9. A voice-driven global control system for intelligent devices, used to implement the method described in any one of claims 1-8, characterized in that, Includes the main body of the device, display screen, voice input component, touch input component, processor, memory, and vibration component; The display screen is located on the front side of the main body of the device and is used to display the active window of the current foreground application and highlight the operation position. The voice input component is disposed on the main body of the device, or disposed on a remote control, an external microphone, or a wearable acquisition device and electrically or communicatively connected to the processor, for the purpose of acquiring user voice. The touch input component is stacked or coupled to the display screen, or disposed on the remote control touch area, touchpad or external pointer input device, and is electrically or communicatively connected to the processor to receive manual touch operations from the user. The vibration component is disposed inside the main body of the device and electrically connected to the processor, and is used to output vibration alerts; The memory stores program instructions that can be executed by the processor. When the processor executes the program instructions, it implements the following functional modules: The voice wake-up and recognition module is used to listen for wake-up words or receive wake-up operations triggered by the user, and convert the user's voice into text commands; The context awareness module is used to obtain the application identifier of the current foreground application, the identifier of the current active window, the accessibility node tree of the current active window, the input method status, and sensitive interface judgment information. The instruction mapping module is used to map the voice intent type to candidate system-level input event parameters based on the text instruction and the interface context obtained by the context awareness module. The permission control module is used to perform sensitive interface judgment, three-channel consistency permission judgment, and security permission matrix judgment on the text instructions, the interface context, and the candidate system-level input event parameters, and to perform one-time execution ticket generation and verification when system-level input events need to be injected; The event injection module is used to inject system-level input events through the user-authorized accessibility service interface or the operating system-permitted input assistance interface after the permission control module has granted permission and the ticket verification has been successfully completed in one go. The post-state verification module is used to determine whether the system-level input event was successfully executed based on the re-acquired interface context and the post-verification conditions after the system-level input event is executed. The security control module is used to enter a security mode when the currently active window is a sensitive interface, allowing only navigation-type security commands to be executed and prohibiting the execution of system-level input events corresponding to swipe intents, click intents, and text input intents; A multimodal feedback module is used to output at least one of voice broadcast, operation position highlighting and vibration prompts after a system-level input event is executed.

10. The voice-driven global control system for intelligent devices according to claim 9, characterized in that, The smart device is any one of the following: smartphone, tablet, smart TV, projector, set-top box, in-vehicle infotainment system, smart cockpit, smartwatch, smart glasses, head-mounted display device, smart speaker with display screen, or smart home control center; When the smart device is a smartphone or tablet, the display screen is a touch screen, and the voice input component is located at the top, bottom, or bezel of the device body; When the smart device is a smart TV, projector, or set-top box, the voice input component is set on the main body of the device, remote control, or external microphone. The event injection module generates corresponding sliding event parameters, directional key event parameters, or focus movement event parameters according to the display resolution or focus layout. When the smart device is an in-vehicle infotainment system or a smart cockpit, the voice input component is located near the center console, ceiling, or steering wheel in the vehicle. The command mapping module is also used to map text commands to system call events corresponding to navigation, music playback, volume adjustment, or air conditioning adjustment.

Images