A voice control method and related device
By acquiring the program identifier and operable component information of voice commands, the user interface of the target application is simulated to automatically operate, solving the problem of frequent eye movement during multiple user interface operations, and realizing the automation and security improvement of voice commands.
Patent Information
- Application Number
- CN202510585759.9
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2025-05-07
- Publication Date
- 2026-11-10
AI Technical Summary
In operation processes involving multiple user interfaces, users need to frequently shift their gaze to the user interface. Especially in scenarios where frequent gaze shifting is not suitable, such as driving a vehicle, existing technologies cannot automate voice command operations, which violates safety principles.
By acquiring the program identifier corresponding to the voice command and information on multiple target operable components, the user interface of the target application can be simulated to automatically operate, reducing the number of times the user's gaze needs to be shifted.
It enables the automatic completion of voice commands for multiple user interfaces without requiring manual operation by the user, reducing the number of times the eyes need to be shifted and improving driving safety.
Smart Images

Figure CN123072385A_ABST
Abstract
Description
Technical Field
[0001] This application relates to the field of natural language processing technology, and in particular to a voice control method and related apparatus. Background Technology
[0002] When operating an application, if the process involves multiple user interfaces, the user needs to frequently shift their gaze to these interfaces to click on corresponding page components, thereby filling in information and navigating between different user interfaces. Taking coffee ordering as an example, the voice command to order a cup of coffee involves the following steps: homepage → product list → product details → filling in delivery information → payment. This means that the number of user interfaces involved in executing this voice command is 5.
[0003] In some application scenarios, it is inconvenient for users to frequently shift their gaze to the user interface, such as in the context of driving a vehicle. If the driver frequently shifts their gaze to the user interface, it would seriously violate driving safety principles.
[0004] Therefore, how to implement voice commands involving multiple user interfaces while reducing the number of times users need to shift their gaze to the user interface is a technical challenge that urgently needs to be solved in this field. Summary of the Invention
[0005] In view of the above problems, this application provides a voice control method and related apparatus to achieve the purpose of implementing voice commands involving multiple user interfaces while reducing the number of times the user needs to shift their gaze to the user interface. The specific solution is as follows:
[0006] The first aspect of this application provides a voice control method, including:
[0007] Get voice commands;
[0008] Obtain the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces;
[0009] Display the user interface of the target application with the program identifier;
[0010] For each user interface of the target application shown, control the operation of the operable component corresponding to the target operable component information in that user interface;
[0011] The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
[0012] In one possible implementation, obtaining information about multiple target operable components corresponding to the voice command includes:
[0013] Obtain keywords from the voice command;
[0014] The target slot type to which the keyword belongs and the target operable component information associated with the keyword in the target slot type are obtained from a preset domain knowledge graph. The domain knowledge graph includes multiple slot types and operable component information corresponding to each of the multiple slot types.
[0015] In one possible implementation, obtaining the program identifier corresponding to the voice command and information on multiple target operable components includes:
[0016] Obtain the target historical voice command that has the highest similarity to the stated voice command;
[0017] From the correspondence between historical voice commands and historical element sets, find the target historical element set corresponding to the target historical voice command. The target historical element set includes the program identifier and information on multiple target operable components used to implement the target historical voice command.
[0018] In one possible implementation, after searching for the target historical element set corresponding to the target historical voice command from the correspondence between historical voice commands and historical element sets, the method further includes:
[0019] The system plays a voice prompt asking the user whether their current needs match the needs of the target historical element set.
[0020] If a voice command with a defined representation is obtained, a first window containing the target historical element set and / or the target historical voice command is displayed.
[0021] In one possible implementation, at least one of the target operable component information is an attribute value of the operable component, and the voice command has a first voice command including the attribute value of the target operable component. Before obtaining the first voice command, the method further includes:
[0022] After controlling the operation of the operable component corresponding to the target operable component information in the user interface, the first operable component in the user interface that is of the interaction type and has not been operated is identified.
[0023] The voice prompt is used to prompt the user to input the attribute value of the first operable component.
[0024] In one possible implementation, the first operable component in the user interface that is of an interaction type and has not been operated includes:
[0025] Obtain the first document object model tree of the user interface, the first document object model tree including the component type of each operable component in the user interface and the attribute value of each operable component, the component type including interactive type and non-interactive type;
[0026] The first operable component that belongs to the interaction type and has not changed is obtained based on the first document object model tree.
[0027] In one possible implementation, the first operable component in the user interface that is of an interaction type and has not been operated includes:
[0028] Obtain a first image containing the user interface of the target application currently being displayed;
[0029] Identify the first operable component in the first image that is in an inactive state.
[0030] In one possible implementation, prior to controlling the operation of the operable component in the user interface that has the operable component information, the method further includes:
[0031] If the target operable component information corresponding to the voice command does not include the operable component information displayed by the user interface, the voice plays the target input prompt information corresponding to the user interface. The target input prompt information is a prompt that prompts the user to input the operable component information displayed by the user interface.
[0032] In one possible implementation, the process of obtaining the target input prompt information corresponding to the currently displayed user interface of the target application includes:
[0033] Obtain information about the operable components in the user interface of the currently displayed target application;
[0034] From a plurality of preset input prompts, obtain the target input prompt information corresponding to the operable component information in the user interface of the currently displayed target application.
[0035] One possible implementation also includes:
[0036] A second window displays the progress information of implementing the voice command in real time. The progress information includes the target input prompt information corresponding to the displayed user interface and the target operable component information corresponding to the target input prompt information. The target input prompt information corresponding to the user interface is a prompt information prompting the user to input the operable component information displayed by the user interface.
[0037] One possible implementation also includes:
[0038] The user interface of the target application is displayed after the operation of the operable components is completed.
[0039] One possible implementation also includes:
[0040] Close the dialog window that obscures the user interface of the target application currently being displayed.
[0041] In one possible implementation, the dialog window that closes the currently displayed user interface of the target application includes:
[0042] Determine if a dialog window exists that is obscuring the user interface of the currently displayed target application;
[0043] Identify the window type of the dialog window;
[0044] If the window type is an authorization type, a voice prompt will be played to remind the user to grant authorization.
[0045] Responding to a voice command containing the authorization result, control the operation of the authorization result in the dialog window;
[0046] Close the dialog window;
[0047] If the window type is unauthorized, close the dialog window.
[0048] One possible implementation also includes:
[0049] Obtain the second document object model tree corresponding to the target application;
[0050] Page navigation relationship information is obtained based on the second document object model tree. The page navigation relationship information includes the navigation relationship of multiple operable components and the position information of the operable components in the user interface.
[0051] One possible implementation also includes:
[0052] If the control operation of the operable component corresponding to the target operable component information in the user interface fails, the first jump path from the user interface in the currently displayed target application to the target user interface is obtained from the page jump relationship information; the target user interface is the user interface that is located after the currently displayed user interface and is closest to the currently displayed user interface and has the operable component corresponding to the target operable information.
[0053] The user interface in the target application is operated sequentially according to the first jump path to control the display of the target user interface.
[0054] In one possible implementation, after controlling the operation of the operable component corresponding to the target operable component information in the user interface, the method further includes:
[0055] Obtain voice commands containing information about operable components to be modified;
[0056] Based on the page jump relationship information, obtain a second jump path from the currently displayed user interface to the user interface with the operable component information to be modified;
[0057] The user interface of the target application is operated sequentially according to the second jump path to control the display of the user interface containing the information of the operable components to be modified.
[0058] One possible implementation also includes:
[0059] If a new operable component is added to the user interface of the target application currently being displayed, a voice prompt will play to inform the user that the new operable component has been added.
[0060] A second aspect of this application provides a voice control device, comprising:
[0061] The first acquisition module is used to acquire voice commands;
[0062] The second acquisition module is used to acquire the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces.
[0063] The first display module is used to display the user interface of the target application with the program identifier;
[0064] The control operation module is used to control the operation of the operable components corresponding to the target operable component information in each user interface of the displayed target application.
[0065] The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
[0066] A third aspect of this application provides a computer program product including computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice control method of the first aspect or any implementation thereof.
[0067] A fourth aspect of this application provides an electronic device, including at least one processor and a memory connected to the processor, wherein:
[0068] The memory is used to store computer programs;
[0069] The processor is used to execute the computer program so that the electronic device can implement the voice control method of the first aspect or any implementation thereof.
[0070] The fifth aspect of this application provides a computer storage medium carrying one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the voice control method described in the first aspect or any implementation thereof.
[0071] By employing the above technical solution, this application provides a voice control method. A user issues a voice command, which corresponds to a program identifier and multiple operable component information representing the functions of operable components. At least two of the operable component information correspond to operable components located in different user interfaces, indicating that multiple user interfaces are involved in implementing the voice command. The method simulates automatic user operation of a target application with a program identifier, thereby launching the target application. For the user interface of the displayed target application, the method simulates user control of the operable components corresponding to the target operable component information contained in the user interface, thus eliminating the need for manual user operation of the target application's user interface and reducing the number of times the user's attention is diverted to the user interface. Attached Figure Description
[0072] The above and other features, advantages, and aspects of the embodiments of this disclosure will become more apparent from the accompanying drawings and the following detailed description. Throughout the drawings, the same or similar reference numerals denote the same or similar elements. It should be understood that the drawings are schematic, and the originals and elements are not necessarily drawn to scale.
[0073] Figure 1 A schematic diagram of a system architecture is provided for this application;
[0074] Figure 2 A flowchart illustrating a voice control method provided in an embodiment of this application;
[0075] Figures 3a to 3b A schematic diagram illustrating one way of displaying a page as provided in an embodiment of this application;
[0076] Figure 4a and Figure 4b A schematic diagram illustrating another implementation of the user interface provided in the embodiments of this application;
[0077] Figures 5a to 5k A schematic diagram illustrating one implementation of the second window provided in an embodiment of this application;
[0078] Figure 6 A schematic diagram of the interface for collapsing user interface 51 and third window 56;
[0079] Figure 7 This is a schematic diagram of the structure of a voice control device provided in an embodiment of this application;
[0080] Figure 8 A schematic diagram of the structure of an electronic device is provided for an embodiment of this application. Detailed Implementation
[0081] The embodiments of this application are described below with reference to the accompanying drawings. The terminology used in the implementation section of this application is for explaining specific embodiments only and is not intended to limit the scope of this application.
[0082] The embodiments of this application will now be described with reference to the accompanying drawings. Those skilled in the art will recognize that, with technological advancements and the emergence of new scenarios, the technical solutions provided in the embodiments of this application are equally applicable to similar technical problems.
[0083] The terms "first," "second," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such terms are interchangeable where appropriate; this is merely a way of distinguishing objects with the same attributes in the embodiments of this application. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, so that a process, method, system, product, or apparatus that comprises a series of elements is not necessarily limited to those elements, but may include other elements not explicitly listed or inherent to those processes, methods, products, or apparatuses.
[0084] The following is an explanation of the terms used in this application.
[0085] HMI (Human-Machine Interaction); MMI (Multimodal Interaction); NUI (Natural User Interface); CUA (Cockpit-Using Agent); COT (Chain of Thought); ISF (Intent and Slot Filling).
[0086] This application can be applied to vehicle-mounted terminals or smart terminals.
[0087] In related technologies, voice control functions can realize basic voice command control, such as navigation route planning, music playback control, and telephone dialing. In other words, voice control functions are voice control for a specific user interface.
[0088] Some applications support a "see-and-talk" mode, which involves labeling media assets with numerical tags. For example, on a navigation map, nearby restaurants, gas stations, and other locations are numbered sequentially according to certain rules, such as "Restaurant No. 1," "Restaurant No. 2," etc. In an audio playlist, each song or radio station is assigned a numerical number, such as "Song 001," "Radio 002," etc. Users can call up or operate the media asset content corresponding to that numerical tag by speaking a voice command containing that numerical tag. In other words, if voice command control is required, the correspondence between media asset content and numerical tags needs to be set.
[0089] Some applications support MindGPT's voice question answering and encyclopedic knowledge query, which means that it can provide users with intelligent question answering functions based on natural language understanding and knowledge base. For example, when users ask common questions such as "What is the weather like today?" or "What is the population of Beijing?", the application can quickly and accurately provide relevant answers by accessing its internal preset knowledge base or connecting with external information sources.
[0090] In summary, in related technologies, voice commands are operations targeting a specific user interface, or they are simple voice questions and answers or knowledge queries, or they include numeric tags. If the correspondence between media asset content and numeric tags is not pre-set for the application, and the voice command is not a simple voice question and answer or knowledge query, and the voice command involves operations on multiple user interfaces, taking coffee ordering as an example, the voice command to order a cup of coffee involves the following operation flow: homepage interface → product list interface → product details interface → fill in delivery information interface → payment interface. That is, the number of multiple user interfaces involved in implementing this voice command is 5. Since related technologies cannot convert voice commands into operation flows to implement them, nor can they determine the multiple user interfaces involved in implementing the voice command through the voice command, users need to frequently shift their attention to the user interface in order to operate it.
[0091] The user interface is a visual interface. Applications in related technologies may adopt a single-threaded task processing mechanism under the layout of GUI (Graphical User Interface) (hereinafter referred to as GUI applications).
[0092] A GUI is a visual interface through which users interact with an application. A GUI includes various page components, such as buttons, menus, text boxes, and icons, arranged in a specific layout on the screen for easy access and manipulation by the user. For example, a music player's GUI might include play / pause buttons, previous / next track buttons, volume control sliders, and a song list display area.
[0093] In GUI applications, a single thread is typically used to handle various tasks and user interactions. This single thread is responsible for listening for and responding to user input events (such as mouse clicks, keyboard presses, touch operations, etc.), updating the content displayed in the user interface, such as navigating from one user interface to another, or updating the property values of page components in the user interface.
[0094] In GUI applications, instructions need to be executed sequentially. For example, when a user clicks on the user interface, such as clicking a button, this click is captured by the GUI application and converted into an event. This event is stored in the GUI application's event queue, awaiting processing by a single thread. The single thread retrieves events from the event queue one by one for processing. For each retrieved click event, the GUI application executes the corresponding instruction according to the predefined event handler. For example, if the user clicks a "play" button, the event handler will call the music playback function to start playing music. After executing the corresponding instructions, the single thread updates the user interface to reflect the result of the operation. For example, after music playback, the music player's user interface might update to display information about the currently playing song, a playback progress bar, etc., to let the user know that the operation has been successfully executed.
[0095] In summary, the relevant technologies require users to confirm each operable component step by step, lacking context-based automated decision-making capabilities.
[0096] In related technologies, voice commands can generally only activate the entry point of a GUI application, such as opening the GUI application. However, it's not possible to perform voice operations on the user interface displayed by the GUI application, such as activating the payment interface of the GUI application via voice commands. Therefore, after users frequently shift their attention to the user interface, they still need to manually operate the interactive components within the user interface (interactive page components are considered interactive page components).
[0097] In certain application scenarios, such as driving, if the driver frequently shifts their gaze to the user interface, it seriously violates driving safety principles. For example, when using the "select items and add to cart" interface, the user needs to repeatedly look at the user interface to select item attributes. Therefore, how to implement voice commands involving multiple user interfaces while minimizing the number of times the user needs to look at the user interface is a pressing technical challenge in this field.
[0098] Based on this, this application provides a voice control method. A user issues a voice command, which corresponds to a program identifier and multiple target operable component information. At least two of the target operable component information correspond to operable components located in different user interfaces, indicating that multiple user interfaces are involved in implementing the voice command. The method simulates automatic user operation of a target application with a program identifier, thereby launching the target application. For the displayed user interface of the target application, the method simulates user control of the operable components corresponding to the target operable component information contained in the user interface, thus eliminating the need for manual user operation of the target application's user interface and reducing the number of times the user's attention is diverted to the user interface.
[0099] The following describes a voice control method and related apparatus provided in the embodiments of this application.
[0100] See Figure 1 , Figure 1 A schematic diagram of a system architecture is shown. The system may include a terminal 100 and a server 200. The server 200 can provide the methods provided in the embodiments of this application to one or more terminals.
[0101] The terminal 100 may have an application installed on it. The application can provide an interface. The terminal 100 can receive relevant parameters input by the user on the interface and send the parameters to the server 200. The server 200 can obtain the processing result based on the received parameters and return the processing result to the terminal 100.
[0102] It should be understood that in some optional implementations, the terminal 100 can also complete the action of obtaining the processing result based on the received parameters on its own, without the need for the server to cooperate. This application embodiment is not limited to this.
[0103] The following description Figure 1 The product form of the mid-terminal 100;
[0104] The terminal 100 in this application embodiment can be a mobile phone, tablet computer, wearable device, vehicle device, augmented reality (AR) / virtual reality (VR) device, laptop computer, ultra-mobile personal computer (UMPC), netbook, personal digital assistant (PDA), etc., and this application embodiment does not impose any restrictions on it.
[0105] Terminal 100 may include a radio frequency unit, memory, input unit, display unit, camera (optional), audio circuitry (optional), speaker (optional), microphone (optional), headphone jack (optional), processor, external interface, power supply, and other components. Those skilled in the art will understand that the above-mentioned components are merely examples and do not constitute a limitation on the terminal or multifunctional device; it may include more or fewer components, or a combination of certain components, or different components.
[0106] The input unit can be used to receive input numeric or character information, and to generate key signal inputs related to user settings and function control of the portable multi-functional device. Specifically, the input unit may include a touchscreen (optional) and / or other input devices. Other input devices may include, but are not limited to, one or more of a physical keyboard, function keys (such as volume control buttons, power buttons, etc.), trackball, mouse, joystick, etc.
[0107] Among them, the input device can receive input data, etc.
[0108] The display unit can be used to display information input by the user or information provided to the user, various menus of the terminal, interactive interfaces, file display, and / or playback of any multimedia file. In the embodiments of this application, the display unit can be used to display interfaces, processing results, etc.
[0109] The memory can be used to store software code related to the voice control method, the processor can execute the steps of the voice control method, and can also schedule other units (such as the above-mentioned input unit and display unit) to achieve the corresponding functions.
[0110] This radio frequency unit (optional) can be used to receive and send signals during information transmission or calls.
[0111] In this embodiment of the application, the radio frequency unit can send data to the server 200 and receive the processing results sent by the server 200.
[0112] It should be understood that this radio frequency unit is optional and can be replaced with other communication interfaces, such as a network port.
[0113] Terminal 100 also includes a power source (such as a battery) for supplying power to the various components.
[0114] Terminal 100 also includes an external interface, which can be a standard Micro USB interface or a multi-pin connector, which can be used to connect terminal 100 to other devices for communication or to connect a charger to charge terminal 100.
[0115] Server 200 includes a bus, a processor, a communication interface, and memory. The processor, memory, and communication interface communicate with each other via the bus.
[0116] The memory can be used to store software code related to instructions from the application, the processor can execute the steps of the chip in response to the instructions from the application, and can also schedule other units to achieve corresponding functions.
[0117] To address the aforementioned problems, this application provides a voice control method. The voice control method of this application embodiment will be described in detail below with reference to the accompanying drawings.
[0118] Reference Figure 2 , Figure 2 This is a flowchart illustrating a voice control method provided in an embodiment of this application, as shown below. Figure 2 As shown in the figure, a voice control method provided in this application embodiment may include steps S201 to S204, which are described in detail below.
[0119] Step S201: Obtain voice commands.
[0120] For example, if this application is applied to an in-vehicle device, the voice command can be monitored after the in-vehicle device is woken up based on the wake word, thereby obtaining the voice command.
[0121] For example, if this application is applied to smart devices, such as smartphones and tablets, voice commands can be monitored after the smart device is unlocked, thereby obtaining the voice commands.
[0122] For example, there can be one or more voice commands. If there are multiple voice commands, the acquisition time of different voice commands can be the same or different; in this application, the acquisition time of different voice commands and the order of execution steps S202, S203 and S204 are not limited, that is, there is no strict order.
[0123] Understandably, voice commands may differ across application scenarios. For example, in a purchase scenario, the voice command might be "order a cup of coffee"; the object could be a beverage, meal, clothing, or ticket. In a work scenario, the voice command might be "send the file you just saved to Zhang San".
[0124] For example, user intent can be recognized through voice commands.
[0125] For example, the voice command could be "Order one iced Americano" or "Order one iced Americano and two iced lattes".
[0126] For example, voice commands may include an SKU identifier.
[0127] In this context, a SKU is the basic unit for measuring inventory inflows and outflows, used to uniquely identify a product or product combination. For example, in a beverage ordering interface, a specific beverage combination (such as medium, hot, Americano) is a SKU. Different combinations of cup sizes, temperatures, and beverage types will generate different SKUs. For instance, SKU1 corresponds to the product combination "medium, hot, Americano," SKU2 corresponds to the product combination "large, iced, latte," and SKU3 corresponds to the product combination "extra-large, room temperature, cappuccino." Voice messages from users can also include SKU identifiers.
[0128] Step S202: Obtain the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces.
[0129] Understandably, there may be multiple applications, so it's necessary to determine which application the voice command is for. Different applications have different application identifiers.
[0130] For example, the program identifier may include, but is not limited to, one or more of the following: the application name, the application package name, and the application signature information.
[0131] For example, an operable component refers to a page component in a user interface that can be triggered by interactive actions (such as clicking, touching, hovering, etc.) to activate specific functions or behaviors.
[0132] For example, operable components include: buttons, links, textboxes, dropdown menus, checkboxes, radio buttons, and sliders.
[0133] For example, operable component information refers to descriptive information about the names of operable components. For instance, in a food delivery application, a user interface includes an "In-Store Pickup" operable component and an "Delivery to Home" operable component. The name of the "In-Store Pickup" operable component is "In-Store Pickup," and the name of the "Delivery to Home" operable component is "Delivery to Home." Operable component information representing the name of the "In-Store Pickup" operable component may include: Pickup, Pickup at Merchant, In-Store Pickup; Operable component information representing the name of the "Delivery to Home" operable component may include: Delivery, Delivery to Home, Mail to Home.
[0134] "At least two of the operable component information are located in different user interfaces," indicating that the implementation of voice commands involves multiple user interfaces and is a relatively complex voice command.
[0135] Step S203: Display the user interface of the target application with the program identifier.
[0136] For example, the CUA agent can be turned on.
[0137] For example, the user interface of the target application is shown as the homepage.
[0138] For example, if it is possible to learn a user's historical actions toward a target application, then the user interface of the target application displayed will be the one with the most frequent historical actions.
[0139] Step S204: For each user interface of the displayed target application, control the operation of the operable component corresponding to the target operable component information in that user interface.
[0140] The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
[0141] For example, the location information of the operable component corresponding to the target operable component information in the user interface can be determined based on the document object model tree of the target application's user interface, thereby achieving the purpose of controlling the operation of the operable component corresponding to the target operable component information in the user interface.
[0142] For example, a screenshot of the user interface of the target application can be obtained; the location information of the operable component corresponding to the target operable component information in the screenshot is identified in the user interface, thereby achieving the purpose of controlling the operable component corresponding to the target operable component information in the user interface.
[0143] The detailed process of step S204 is described below, which includes steps A1 to A4.
[0144] Step A1: The operable components corresponding to the target operable component information contained in the user interface displayed in the control operation step S203.
[0145] Understandably, if the target application is a GUI application, events controlling the operation of operable components can be stored in the GUI application's event queue, awaiting processing by a single thread. The single thread retrieves events from the event queue sequentially for processing. For these events, the GUI application executes corresponding instructions according to predefined event handlers. After executing the instructions, the single thread updates the user interface to reflect the result of the operation.
[0146] The process of updating the user interface in a single thread may involve updating the property values of operable components in the user interface, such as updating the property value of the operable component "temperature" to "ice".
[0147] The process of updating the user interface in a single thread may involve jumping from the current user interface to another user interface, for example, jumping from the "Meal Pickup Method Selection Interface" to the "Store Selection Interface".
[0148] Step A2: Control the operable components corresponding to the target operable component information contained in the currently displayed user interface, return to step A2, until the voice command is implemented.
[0149] The embodiments of this application can realize cross-page operation chain arrangement, that is, automatically jump between user interfaces.
[0150] As time goes by, the currently displayed user interface may remain the same or it may change.
[0151] Understandably, the multiple user interfaces of the target application are designed to implement voice commands. Taking coffee ordering as an example, the voice command is "order a coffee." The operation process to implement this voice command includes: home screen → product list screen → product details screen → fill in delivery information screen → payment screen. The target application displays the payment screen, and after the payment is completed, the voice command "order a coffee" is implemented.
[0152] For example, the type of user interface displayed in different application scenarios may be the same or different; for example, in the application scenario of purchasing objects, the type of user interface displayed in the end is a payment interface.
[0153] For example, the type of the final user interface displayed can be determined based on natural language processing technology.
[0154] For example, the type of user interface to be displayed at the end can be preset for different application scenarios.
[0155] This application proposes mapping voice commands to operations of multiple user interfaces within an application, without requiring API (Application Programming Interface) dependencies, and automating the operation of operable components through DOM tree parsing or image recognition.
[0156] The embodiments of this application use voice-led and minimal visual feedback to reduce the time required for eye movement to the user interface in the target application.
[0157] This application provides a voice control method. A user issues a voice command, which corresponds to a program identifier and multiple operable component information representing the functions of operable components. At least two of the operable component information correspond to operable components located in different user interfaces, indicating that multiple user interfaces are involved in implementing the voice command. The method simulates automatic user operation of a target application with a program identifier, thereby launching the target application. For the user interface of the displayed target application, the method simulates user control of the operable components corresponding to the target operable component information contained in the user interface, thus eliminating the need for manual user operation of the target application's user interface and reducing the number of times the user's attention is diverted to the user interface.
[0158] It is understood that there are multiple ways to implement step S202, and the embodiments of this application provide, but are not limited to, the following two.
[0159] The first method of implementing step S202 includes the following steps B11 to B12.
[0160] In the first implementation, there can be one or more voice commands, and the acquisition time for different voice commands may be the same or different.
[0161] For example, if a user wants to buy a mocha, their voice command might be, "Buy a large mocha with ice from mini-program A, self-pickup, password-free payment." This voice command contains all the actionable components needed to achieve the desired result, such as "a cup," "ice," "large," "mocha," "self-pickup," and "password-free payment." The user doesn't need to input the voice command again. In this scenario, the number of voice commands is one. In this case, a single sentence can trigger the entire process, reducing the number of steps.
[0162] For example, if a user wants to buy a mocha, and their voice command is "Buy a mocha," then subsequent voice commands will guide the user to issue commands containing the "program identifier," "one," "ice," "large," "self-pickup," and "password-free payment." These voice commands do not need to be issued simultaneously. In this scenario, there are multiple voice commands.
[0163] If the number of voice commands is 1, then the user does not need to manually confirm the information of each target operable component; the process can be automated.
[0164] Step B11: Obtain keywords from the voice command.
[0165] For example, the number of keywords in a voice command can be one or more.
[0166] For example, speech data can be acquired using a microphone array and subjected to environmental noise suppression processing; speech endpoint detection can be performed on the speech data to obtain speech commands.
[0167] Microphone arrays are installed in suitable locations according to a certain geometric arrangement. Taking a vehicle as an example, microphones can be located in rearview mirrors, dashboards, etc. Microphone arrays are used to capture voice data and convert the voice data into electrical signals.
[0168] The environment in which a user issues voice commands may contain various background noises. If a user issues voice commands while driving, the background noise includes, but is not limited to: engine noise, tire friction noise, wind noise, and conversations between passengers. The purpose of environmental noise suppression is to reduce the interference of background noise on voice commands and improve the clarity of voice commands.
[0169] The main task of speech endpoint detection is to accurately determine the start and end points of speech commands, that is, to distinguish between speech activity segments and non-speech segments (such as silent or pure noise segments).
[0170] For example, this application addresses the multilingual recognition of voice commands. For instance, multiple languages can be recognized based on deep learning techniques. For example, feature representations of voice commands in different languages can be learned based on recurrent neural networks (RNNs) and their variants (such as Long Short-Term Memory networks (LSTM) and gated recurrent units (GRUs)) and the Transformer architecture.
[0171] For example, voice commands can be converted into text, and then keywords can be extracted from the text.
[0172] Step B12: Obtain the target slot type to which the keyword belongs and the target operable component information associated with the keyword in the target slot type from the preset domain knowledge graph. The domain knowledge graph includes multiple slot types and operable component information corresponding to the multiple slot types respectively.
[0173] Steps B11 to B12 are slot classification processes based on domain knowledge graphs.
[0174] In a domain knowledge graph, slot types are categories used to define and classify information about various operable components related to the corresponding domain.
[0175] The following example illustrates the concept of domain knowledge graphs.
[0176] If this application is applied to a beverage purchasing scenario, the domain knowledge graph includes multiple slot types as follows: cup type, temperature, beverage type, quantity, store, and pickup method. The operable component information corresponding to the slot type "cup type" includes: medium, large, extra-large; the operable component information corresponding to the slot type "temperature" includes: hot, room temperature, ice; the operable component information corresponding to the slot type "beverage type" includes: latte, cappuccino, mocha, ...; the operable component information corresponding to the slot type "store" includes: store A, store B, ...; and the operable component information corresponding to the slot type "pickup method" includes: in-store pickup, home delivery.
[0177] In the domain knowledge graph, the slot type is entity. The following explains the edges between entities.
[0178] For example, if the selection of one slot type is related to the selection of another slot type, then there is an edge between the two slot types. For instance, some beverage types may have specific cup size restrictions. For example, extra-large cups may not be available for certain beverage types.
[0179] For example, if the selection of one slot type requires the selection of another slot type, then there is an edge between the two slot types. For instance, there is a dependency relationship between the pickup method and the store, because some stores do not support the "delivery to home" pickup method.
[0180] For example, if the slot types have a certain order in the operation process, such as selecting the pick-up method before selecting the store, and selecting the cup type, temperature, beverage type, and quantity after selecting the store, then these slot types have edges between them.
[0181] Understandably, the keywords in a user's voice command may not perfectly match the target actionable component information. For example, the target actionable component information might be "pick up in-store," while the corresponding keyword might be "pick up your own food"; or the target actionable component information might be "Americano," while the corresponding keyword might be "energize." Based on this, the fuzzy semantic parsing engine can be used to perform fuzzy semantic parsing on the keywords.
[0182] A fuzzy semantic parsing engine is a technology designed to handle fuzziness and polysemy in natural language. It analyzes and processes keywords in voice commands and maps them to target operable component information of the corresponding target slot type.
[0183] For example, a thesaurus or near-synonym dictionary can be built. The thesaurus includes keywords with the same meaning. For instance, "pick up your own food" and "pick up in store" have the same meaning. If the keyword in the voice command is "pick up your own food," it can be mapped to the operable component information "pick up in store" corresponding to the slot type "food pickup method."
[0184] The first method to implement step S202 is to dynamically bind the intent and keywords of the voice command with the operable component information of the operable component of the user interface in the target application, thus realizing the tripartite mapping of intent-slot type-operable component information.
[0185] The second method for implementing step S202 includes the following steps B21 to B22.
[0186] Step B21: Obtain the target historical voice command with the highest similarity to the stated voice command.
[0187] For example, a voice command may include one or more keywords, with each keyword corresponding to a target operable component information. That is, the target operable component information corresponding to the voice command may be part of the operable component information or all of the operable component information that implements the voice command.
[0188] For example, the voice command "Order a latte" includes the two keywords "a cup" and "latte." The complete set of operable components that implement this voice command includes: "a cup," "latte," one operable component for the slot type "temperature" (e.g., "room temperature"), one operable component for the slot type "cup size" (e.g., "large"), one operable component for the slot type "pickup method" (e.g., "in-store pickup"), and one operable component for the slot type "store" (e.g., "Beijing Xindongan Store"). The target operable component information corresponding to the above voice command represents a portion of the operable components that implement the voice command.
[0189] For example, the operable component information corresponding to the target historical voice command is at least part of the operable component information that implements the target historical voice command. For instance, the target historical voice command could be "order a latte".
[0190] Step B22: From the correspondence between historical voice commands and historical element sets, find the target historical element set corresponding to the target historical voice command. The target historical element set includes the program identifier and information on multiple target operable components used to implement the target historical voice command.
[0191] Related technologies cannot remember user preferences such as target historical voice commands. This application can store historical voice commands to simplify the interaction process and avoid repetitive operations.
[0192] This application can automatically obtain the target slot type and the target operable component information associated with the keyword based on the historical element set, thereby reducing the number of interaction steps.
[0193] For example, the target set of historical elements may also include payment methods; for example, payment methods include verification methods and payment channels.
[0194] For example, verification methods include password-free payment, fingerprint payment, facial recognition payment, and password payment.
[0195] For example, a payment channel refers to the payment method that a user can choose when making a payment. It connects users, merchants, and financial institutions to complete the transfer of monetary funds.
[0196] It is understood that the voice control method provided in this application embodiment has already been executed for historical voice commands, meaning that the historical element set corresponding to the historical voice commands contains all target operable component information and program identifiers that implement the historical voice commands. If there is a target historical voice command with a high degree of similarity to the voice command, the operable component information in the target historical element set corresponding to the target historical voice command can be grouped into the target operable component information corresponding to the voice command. The program identifier in the target historical element set can be used as the program identifier of the target application to be displayed now. This eliminates the need for repeated voice prompts to the user to input target operable component information.
[0197] For example, a voice prompt can be played to ask the user whether their current need matches the needs of the target historical element set; if the user issues a voice command indicating "confirmation", steps S203 to S204 can be executed. If the user issues a voice command indicating "negation", step S202 can be executed in the first implementation method.
[0198] For example, even without a voice prompt asking the user whether their current need matches the needs of the target historical voice command, steps S203 to S204 can be executed, and finally, a prompt representing all target operable component information can be played, such as "Please confirm whether you want to order an iced extra-large latte for in-store pickup." If the user needs to modify the information of a target operable component, the voice command can be issued again; if the user does not need to modify it, a voice command representing "confirm" can be issued.
[0199] Understandably, a first window containing the target's historical elements can also be displayed. This allows the user to view all the target operable components that implement the voice command with a single glance at the user interface, thus reducing the number of times the user needs to look at the user interface. Based on this, after step B22, this application further includes steps C1 to C2.
[0200] Step C1: Play a voice prompt asking the user whether their current needs match the needs of the target historical element set.
[0201] Step C2: If a voice command with a defined representation is obtained, display a first window containing the target historical element set and / or the target historical voice command.
[0202] To help those skilled in the art better understand the methods provided in the embodiments of this application, examples are given below.
[0203] The following explanation uses the vehicle application as an example.
[0204] like Figures 3a to 3b The diagram shown is a schematic representation of one way a page is displayed according to an embodiment of this application.
[0205] Assuming the user wakes up the in-vehicle device using a wake word and issues the voice command "Give me an iced Americano," then the following can be displayed: Figure 3a The page shown displays capsule 31 containing text corresponding to voice commands in the user interface.
[0206] The in-vehicle device plays a voice prompt asking the user whether their current need matches the needs of the target historical element set. If a definite voice command is received, it is displayed. Figure 3b The user interface shown.
[0207] like Figure 3b As shown, assuming the target historical voice command with the highest similarity to the voice command "Give me an iced Americano" is "Order an iced Americano from program identifier A", the set of target historical elements corresponding to this target historical voice command includes: program identifier A, self-pickup, Beijing Xindongan store, one cup, large, Americano, and password-free payment. Figure 3b The first window in the first window 32 includes: the target historical voice command "Order an iced Americano from program identifier A" and the target historical element set corresponding to the target historical voice command: "#program identifier A#self-pickup#Beijing Xindongan store#one cup#large#Americano#password-free payment". For example, different elements can be separated by preset identifiers. Figure 3b This explanation uses the default identifier "#" as an example.
[0208] For example, during the process of operating the user interface based on the target historical element set, if the target operable component information corresponding to the voice command issued by the user conflicts with the operable component information in the target historical element set, the target operable component information corresponding to the voice command shall prevail.
[0209] Understandably, if the voice command is an operation related to a specific object, such as a purchase operation, then the attribute value of the object's slot type needs to be selected. Taking "drink" as an example, the attribute values of the quantity, cup type, temperature, and drink type need to be selected. If an attribute value of a certain slot type has not been operated, then a prompt message needs to be played to prompt the user to input the attribute value of the operable component.
[0210] For example, at least one target operable component information is the attribute value of the operable component, the voice command has a first voice command including the attribute value of the target operable component, and before obtaining the first voice command, the following steps D1 to D2 are further included.
[0211] Step D1: After controlling the operation of the operable component corresponding to the target operable component information in the user interface, identify the first operable component in the user interface that belongs to the interaction type and has not been operated.
[0212] For example, the number of first operable components can be one or more.
[0213] For example, there are multiple ways to implement step D1, and the embodiments of this application provide, but are not limited to, the following two.
[0214] The first method of implementing step D1 includes steps D111 to D112.
[0215] Step D111: Obtain the first document object model tree of the user interface. The first document object model tree includes the component type of each operable component in the user interface and the attribute value of each operable component. The component type includes interactive type and non-interactive type.
[0216] The Document Object Model Tree (DOM) is a tree-like data structure generated after parsing the user interface, used to represent the content and structure of the user interface. Manipulable components include, but are not limited to: element nodes (each element node can have multiple child nodes), attribute nodes (representing the attributes of element nodes, such as id, class, attribute values, etc.), text nodes (representing the text content in element nodes), comment nodes (representing the comment content in the document), and user interface nodes (the root node of the DOM, representing the entire user interface).
[0217] It is understandable that after controlling the operation of the operable component corresponding to the target operable component information in the user interface, the attribute values of the operated target operable component in the first document object model tree change, still based on... Figure 4a and Figure 4b For example, after controlling the operable component corresponding to the target operable component information in the user interface, the attribute value of "Quantity" of the operable component corresponding to the target operable component information "One Cup" changes to "1", the attribute value of "Temperature" of the operable component corresponding to the target operable component information "Ice" changes to "Ice", and the attribute value of "Drink Type" of the operable component corresponding to the target operable component information "Americano Coffee" changes to "Americano Coffee". However, the attribute value of the operable component "Cup Type" remains unchanged.
[0218] Step D112: Obtain the first operable component that belongs to the interaction type and whose attribute values have not changed based on the first document object model tree.
[0219] The second method for implementing step D1 includes steps D121 to D122.
[0220] Step D121: Obtain a first image containing the user interface of the target application currently being displayed.
[0221] For example, a screenshot can be taken of the currently displayed user interface.
[0222] Step D122: Identify the first operable component in the first image that is in an inactive state.
[0223] For example, OCR (Optical Character Recognition) technology can be used to identify the text in the first image, and based on the relationships between the text in the first image, the operable components in the first image and their corresponding attribute values can be determined. For example, the attribute values corresponding to the operable component "temperature" are "hot", "room temperature", and "ice".
[0224] For example, the pixel value of an attribute in an inactive state is different from the pixel value of an attribute in an active state. Figure 4a As shown, it will not be elaborated further here.
[0225] For example, the attribute value corresponding to quantity might consist of increase / decrease buttons and a number display. If the quantity is displayed as the default value (such as "0"), it is considered that the "quantity" is in an inactive state.
[0226] For example, some operable components correspond to multiple attribute values, but these multiple attribute values are mutually exclusive. For instance, the operable component "temperature" corresponds to three attribute values: hot, room temperature, and ice. Only one of the three attribute values of the operable component "temperature" can be operated. If any one of the three attribute values of the operable component "temperature" is in an operated state, then the operable component is in an operated state.
[0227] Step D2: Play a voice prompt message to prompt the user to input the attribute value of the first operable component.
[0228] For example, a user-product association matrix can be constructed, whereby the product association matrix includes the frequency of operation of attribute values corresponding to operable components of a product, thereby determining the probability of a user selecting each attribute value corresponding to an operable component. This application embodiment can automatically obtain the attribute values of the first operable component based on the product association matrix. For example, if user A has the highest probability of selecting a large size when purchasing a beverage, then the attribute value of the first operable component "cup type" is "large".
[0229] To help those skilled in the art better understand the technical solutions involved in steps D1 to D2 above, examples are provided below. For instance... Figures 4a to 4b The diagram shown is a schematic representation of another implementation of the user interface provided in the embodiments of this application.
[0230] like Figure 4a As shown, if the target operable component information corresponding to the voice command includes "Americano," "Ice," and "One Cup," then the operable components "Drink Type" corresponding to the target operable component information "Americano," "Temperature" corresponding to the target operable component information "Ice," and "Quantity" corresponding to the target operable component information "One Cup" in the control operation user interface 41 are configured. The three operable components "Temperature," "Quantity," and "Drink Type" in the user interface 41 have been configured with corresponding attribute values, and these three operable components are in an operated state. For example, the configured attribute values are in an operated state; the pixel values of attribute values in an operated state are different from those in an unoperated state, such as... Figure 4aAs shown, the attribute value "ice" is in an manipulated state, while the attribute values "medium", "large", and "extra-large" are all in an inactive state. Therefore, the operable component "cup shape" is in an inactive state.
[0231] pass Figure 4a It can be seen that the attribute value of "cup type" has not been manipulated, while the attribute values of "quantity", "drink type" and "temperature" have all been manipulated. Therefore, the first operable component in the user interface 41 that belongs to the interaction type and is in an unmanaged state is "cup type", so the prompt "Please enter the cup type" can be played via voice.
[0232] As shown in Figure 4, if the user's voice command includes "large", the capsule 31 will display the text "large" of the voice command. Then, based on "large", the user interface 41 will be operated to obtain... Figure 4b The user interface shown. Figure 4b The "cup shape" in the user interface 41 is in an activated state.
[0233] It is understood that before executing step S204, the method further includes: if the target operable component information corresponding to the voice command obtained in step S201 does not include the operable component information displayed by the user interface, the input prompt information corresponding to the user interface is played by voice, and the input prompt information is a prompt message prompting the user to input the operable component information displayed by the user interface.
[0234] by Figure 3b For example, the currently displayed user interface is the "food pickup method selection interface". If the target operable component information corresponding to the voice command obtained in step S201 does not include "in-store pickup" and "delivery to home", then the input prompt information corresponding to the "food pickup method selection interface" needs to be displayed. For example, the input prompt information can be "select food pickup method" or "whether you want in-store pickup or delivery to home".
[0235] It is understood that there are multiple methods for obtaining the target input prompt information corresponding to the currently displayed user interface. The embodiments of this application provide, but are not limited to, the following two methods.
[0236] The first process for obtaining target input prompt information corresponding to the user interface of the currently displayed target application includes the following steps E11 to E12.
[0237] Step E11: Determine the target order of the currently displayed user interface of the target application in the previously displayed user interfaces of the target application.
[0238] The following example illustrates this. Suppose that the user interface of the target application currently displayed is the "Select Items and Add to Cart" interface, and the user interfaces of the target application already displayed include the "Meal Pickup Method Selection" interface and the "Store Selection" interface. Then, the target order of the user interfaces of the target application currently displayed is 3.
[0239] Step E12: Determine the target input prompt information as the input prompt information whose execution order is the same as the target order among the preset multiple input prompt information.
[0240] For example, the execution order of multiple input prompts can be preset; the execution order of the corresponding input prompts is the same as the target order of the user interface.
[0241] It is understandable that different applications of the same type may have different or the same user interfaces when implementing the same voice command. Based on this, the embodiments of this application extract the commonalities of different applications of the same type. For example, different applications of the type "buy drinks" may have multiple user interfaces when responding to the same voice command, including: a pick-up method selection interface, a store selection interface, a product selection and addition to the shopping cart interface, a payment interface, and a drink preparation progress interface.
[0242] It is understandable that the functions of operable components displayed in the user interface of different applications of the same type may be the same or different. For example, in some applications, the "food pickup method selection interface" includes two operable components: "in-store pickup" and "delivery to home"; in other applications, the "food pickup method selection interface" includes four operable components: "in-store pickup", "delivery to home", "drive-thru pickup", and "dine-in".
[0243] It is understandable that different applications of the same type may have the same or different order of appearance of multiple user interfaces when implementing the same voice command. For example, some applications may implement the voice command "order an iced Americano" in the following order: order selection interface → store selection interface → select item and add to cart interface → order checkout interface → drink preparation progress interface; while other applications may implement the same voice command "order an iced Americano" in the following order: store selection interface → order selection interface → select item and add to cart interface → order checkout interface → drink preparation progress interface.
[0244] In summary, in the first implementation method, the execution order of the multiple input prompts is fixed, which may result in the target input prompts played by the voice not corresponding to the user interface. Based on this, the embodiments of this application provide a second implementation method.
[0245] The second method for obtaining the target input prompt information corresponding to the user interface of the currently displayed target application includes the following steps E21 to E22.
[0246] Step E21: Obtain information about the operable components in the currently displayed user interface.
[0247] For example, an image of the user interface is obtained; text in the image is recognized using OCR technology; and the text corresponding to the information of the operable components is identified.
[0248] For example, the document object model tree of the user interface can be obtained, thereby obtaining information about the operable components in the currently displayed user interface.
[0249] Step E22: From a set of preset input prompts, obtain the target input prompt information corresponding to the operable component information in the user interface of the currently displayed target application.
[0250] For example, the similarity between the information of operable components in the currently displayed user interface and the input prompt information can be calculated, and the input prompt information with the highest similarity can be determined as the target input prompt information. That is, the order in which multiple input prompt information are played via voice is variable.
[0251] In an optional implementation, this embodiment further includes: displaying the user interface of the target application after the operation of the operable component is completed. See also... Figure 5k and Figure 5k The description of the user interface of the target application after the operation of the operable components is completed will not be repeated here.
[0252] It is understood that, in order to facilitate users in viewing the progress of implementing voice commands, this application embodiment also provides the following method: displaying a second window to show the progress information of implementing the voice command, the progress information including target input prompt information corresponding to the displayed user interface and target operable component information corresponding to the target input prompt information, wherein the target input prompt information corresponding to the user interface is a prompt information prompting the user to input the operable component information displayed by the user interface.
[0253] For example, in addition to displaying progress information through a second window, progress information can also be displayed through status icons. That is, operation details are not displayed unless necessary, and progress is only displayed through status icons such as progress bars, thereby minimizing visual distraction.
[0254] To help those skilled in the art better understand the second window provided in the embodiments of this application, examples are given below.
[0255] For example, the first window and the second window can be the same window or different windows.
[0256] For example, the user interface of the target application and the first window / second window have no overlapping area; for example, the first window / second window may float above the user interface of the target application.
[0257] like Figures 5a to 5k The diagram shown is a schematic representation of one implementation of the second window provided in an embodiment of this application.
[0258] The following is combined Figures 5a to 5k An example of a voice control method is given, which includes the following steps F1 to F10.
[0259] Step F1: Obtain the voice command "Order an iced Americano".
[0260] like Figure 5a As shown, assuming that after waking up the in-vehicle device based on the wake word, the user issues the voice command "order an iced Americano", the text corresponding to the voice command can be displayed in Capsule 31.
[0261] Step F2: Play a voice prompt to indicate which application to launch; if a voice command containing the application identifier is received, display the target application with the application identifier.
[0262] like Figure 5b As shown, before the user issues a voice command containing the program identifier, it can be displayed Figure 5b The user interface shown can display "Listening," indicating that it is waiting for the user to input a voice command containing the program identifier. If the user issues a voice command containing the program identifier, then the following can be displayed: Figure 5c The user interface shown here allows the "program identifier" to be displayed in capsule 31.
[0263] like Figure 5d As shown, a second window 51 can be displayed while the target application is being displayed; the second window 51 displays the voice command "order an iced Americano" and the target operable component information corresponding to the voice command: "one", "ice", "Americano".
[0264] like Figure 5dThe image shows the homepage 52 of the target application, which is the "Food Pickup Method Selection Interface" 52. At this time, the second window will display the input prompt information corresponding to the "Food Pickup Method Selection Interface": "Select Food Pickup Method" or "Please select in-store pickup or home delivery".
[0265] Since the voice command in step F1 does not correspond to the operable component information displayed on the "Meal Pickup Method Selection Interface", it is necessary to play the prompt information by voice.
[0266] Step F3: Play the voice prompt message "Select Pickup Method" or "Please select in-store pickup or home delivery" corresponding to the "Pickup Method Selection Interface"; if the voice command includes "I want to pick up in-store" is obtained; control the operation of the operable component "In-store Pickup" in the "Pickup Method Selection Interface"; control the second window to display "In-store Pickup" corresponding to the input prompt message "Select Pickup Method".
[0267] Understandably, the target actionable component information corresponding to the keyword "self-pickup" is "in-store self-pickup".
[0268] like Figure 5e As shown, the second window 51 displays input prompts for "Select Pickup Method" and "Pickup at Store".
[0269] Step F4: The voice prompt "Select a self-pickup store" is played; and the input prompt message "Select a self-pickup store" is displayed on the second window corresponding to the "Store Selection Interface".
[0270] For example, embodiments of this application can automatically obtain information about operable components not included in voice commands through environmental parameter fusion. For instance, by combining vehicle sensor data, such as GPS (Global Positioning System) location, the operable component information of the "store" can be automatically determined to be the store closest to the vehicle.
[0271] Understandably, since the voice commands already obtained do not correspond to any operable components related to "stores", it is necessary to play voice prompts.
[0272] like Figure 5f As shown, after the operable component "Pick Up in Store" is selected, the target application jumps from the "Pickup Method Selection Interface" to the "Store Selection Interface" 53. At this time, the second window also displays the input prompt information "Select Pickup Store" corresponding to the "Store Selection Interface".
[0273] Step F5: Obtain a voice command containing "Beijing Xindongan Store" and control the operable component "Program Identifier (Beijing Xindongan Store)" in the "Store Selection Interface"; control the second window to display the input prompt message "Select Self-pickup Store" corresponding to "Beijing Xindongan Store".
[0274] like Figure 5g As shown, after obtaining the voice command containing "Beijing Xindongan Store", the text "Beijing Xindongan Store" can be displayed in Capsule 31.
[0275] Understandably, after controlling the operable component "Program Identifier (Beijing Xindongan Store)" in the "Store Selection Interface", one can jump from the "Store Selection Interface" to the "Select Products and Add to Cart Interface" 41. Therefore, the second window can also display the input prompt information "Select Products and Add to Cart" corresponding to the "Select Products and Add to Cart Interface" 41.
[0276] It is understandable that the voice command has already obtained information on some operable components in the "Select Products and Add to Cart" interface 41, so the "Select Products and Add to Cart" interface 41 is operated first.
[0277] Step F6: Control the "Beverage Type" attribute value to "Americano Coffee", the "Temperature" attribute value to "Ice", and the "Quantity" attribute value to 1 in the operation "Select Products and Add to Cart" interface 41; control the second window to display the input prompt information "Select Products and Add to Cart" corresponding to "Americano Coffee", "Ice", and "One Cup".
[0278] like Figure 5h As shown, this is the "Select Products and Add to Cart" interface after step F6.
[0279] Step F7: Identify the first interactive component "cup shape" in the "Select Products and Add to Cart" interface 41 that is an interactive type but has not been operated. Play a voice prompt message to encourage the user to input the attribute value of the first interactive component.
[0280] Step F8: Obtain the voice command containing "large", control the attribute value of the operable component "cup type" in the operation "select product and add to shopping cart interface" 41 to be "large"; control the second window to display the "large" corresponding to the input prompt message "select product and add to shopping cart".
[0281] like Figure 5i As shown, this is the "Select Items and Add to Cart" interface 41, where the operable component "Cup Type" has the attribute value "Large" after the user control operation "Select Items and Add to Cart" is set to "Large".
[0282] Step F9: The input prompt "Order Settlement" corresponding to the "Payment Interface" is played via voice; if a voice command indicating "Confirm" is received, the payment operation is performed; the second window is controlled to display the input prompt "Order Settlement" and the corresponding payment method.
[0283] It is understandable that since the voice command does not correspond to any operable component with a payment interface, it is necessary to play "order settlement" via voice.
[0284] For example, payment methods include verification methods and payment channels, wherein the verification methods include, but are not limited to, any one of: password-free payment, fingerprint payment, facial recognition payment, and password payment.
[0285] like Figure 5j The image shown is a schematic diagram of the payment interface 54 after confirming the password-free payment.
[0286] Step F10: Display the "Drink Preparation Progress Interface" 55.
[0287] Understandably, after payment is completed, the target application can redirect to the "drink preparation progress screen" 55.
[0288] like Figure 5k The "drink preparation progress interface" 55 and the third window 56 are shown. For example, to facilitate users viewing the drink preparation progress, a third window 56 can also be displayed. The third window 56 displays the drink preparation progress, allowing users to quickly view the progress by simply looking at the third window 56. This reduces the time users spend shifting their attention to the target application's user interface.
[0289] For example, the user can collapse the user interface 51 and the third window 56, or, after a preset interval, the user interface 51 and the third window 56 can be automatically collapsed; Figure 6 The diagram shown is a schematic of the interface where the user interface 51 and the third window 56 are collapsed.
[0290] like Figure 6 As shown, it can be done through Figure 6 The button 61 shown collapses or expands the user interface 51 and the third window 56. Alternatively, the user interface 51 and the third window 56 can also be collapsed or expanded via voice commands.
[0291] For example, Figure 5k and Figure 5j All of these are the user interfaces of the target application described after the operation of the operable components is completed.
[0292] As demonstrated by steps F1 to F10, this application bridges the semantic gap between natural language commands and the internal operations of the target application, automatically decomposing the user's spoken voice commands into atomic, operable component information. This transforms the operational flow of implementing voice commands into an atomic sequence of operations. No layer-by-layer voice confirmation is required unless the target operable component information corresponding to the user's issued voice command is incomplete, in which case the user needs to be prompted to input the corresponding voice command.
[0293] For example, the voice control method provided in this application embodiment can perform voice confirmation only for key steps, such as voice commands to confirm payment and prohibit payment for the "payment interface".
[0294] For example, the voice control method provided in this application embodiment can perform voice confirmation after each user interface control operation. For instance, after the control operation "Food Pickup Method Selection Interface", the system can play a voice prompt asking "Confirm whether to pick up in-store", and the user's voice command is "Confirm", thus performing voice confirmation.
[0295] pass Figures 5a to 5k It is understood that the voice control method provided in this application can perform multimodal interactive verification channels, such as playing voice prompts, displaying key target operable component information in a first or second window (e.g., Figure 5k As shown), the target application's user interface highlights information about the target's operable components (such as...). Figure 5i (As shown).
[0296] For example, this application can also define lightweight interaction rules that are voice-led and vision-assisted, and dynamically control the feedback granularity through confidence thresholds. For instance, after controlling the user interface of the target application, if the confidence of "controlling the user interface of the target application" is high, then no voice confirmation is required; if the confidence of "controlling the user interface of the target application" is low, then voice confirmation is required.
[0297] For example, embodiments of this application also provide a safety classification strategy, which limits the complexity of interaction based on vehicle speed (e.g., stationary / low speed / high speed), such as prohibiting manual parameter adjustment when driving at high speed.
[0298] Understandably, related technologies lack automatic handling logic for abnormal scenarios such as pop-ups and permission requests, which require manual authorization, as is the case during payment. That is, during the display of the target application's user interface, the target application may display other dialog windows besides the user interface, such as advertising windows or authorization windows. These dialog windows may obscure at least a portion of the target application's displayed user interface; for example, they may obscure operable components of the target application's displayed user interface, potentially causing failure to control the operable components corresponding to the target operable component information within that user interface. Based on this, embodiments of this application provide the following method: closing the dialog window obscuring the currently displayed user interface of the target application.
[0299] Understandably, closing some dialog windows does not affect operable components in the user interface, such as advertising windows; however, closing others may affect operable components. For example, in the "store selection interface," if the user's voice command is "select the nearest store," then "allow access to geolocation" authorization is required. In this case, a dialog window containing "Allow access to geolocation? Allow only, Always allow, Deny" can be displayed. If this dialog window is closed directly, store selection becomes impossible, meaning the user interface displayed by the target application cannot be controlled. Therefore, dialog windows need to be differentiated: if it's an authorization window, a voice prompt with authorization information should be played; if it's an advertising window, it can be closed directly.
[0300] Based on this, the implementation method of "closing the dialog window that obscures the user interface of the currently displayed target application" provided in the embodiments of this application includes the following steps G1 to G6.
[0301] Step G1: Determine if there is a dialog window that obscures the user interface of the currently displayed target application.
[0302] For example, a screenshot of the user interface of the currently displayed target application can be acquired in real time; the target pre-stored image corresponding to the user interface identifier of the currently displayed target application can be found from the correspondence between user interface identifiers and pre-stored images; the area in the screenshot that differs from the target pre-stored image is determined to be the area where the dialog window is located. For example, the pre-stored image is a screenshot of the user interface of the target application obtained without any dialog window being displayed.
[0303] Step G2: Identify the window type of the dialog window.
[0304] For example, step G2 can be implemented by: acquiring an image of the dialog window; recognizing text information in the image; and determining the window type of the dialog window based on the text information.
[0305] For example, if the text information includes authorization, the dialog window's window type is authorized. If the text information does not include authorization, the dialog window's window type is unauthorized, such as an advertising window.
[0306] Step G3: If the window type is an authorization type, a voice prompt message is played to remind the user to grant authorization.
[0307] For example, the prompt message used to ask the user for authorization may include all the information displayed in the dialog window, so that the user can determine what to authorize.
[0308] Step G4: Respond to the voice command containing the authorization result, and control the operation of the authorization result in the dialog window.
[0309] For example, the authorization result can be any of the following: prohibit, allow when used, or always allow.
[0310] Step G5: Close the dialog window.
[0311] Step G6: If the window type is unauthorized, close the dialog window.
[0312] For example, unauthorized dialog windows include, but are not limited to: advertising windows and login windows.
[0313] It is understood that the voice control system in which the voice control method provided in this application is located is often independent of the target application. The attribute values of certain operable components in the user interface of the target application may change; for example, the inventory of a certain beverage will gradually decrease over time if it is not replenished. If the voice control system cannot promptly detect changes in the attribute values of certain operable components, it may be unable to execute the voice command. For instance, if the user issues the voice command "Buy promotional item A," but the inventory of promotional item A is zero, the voice command "Buy promotional item A" cannot be executed.
[0314] Based on this, the embodiments of this application propose the following method: if the inventory of the operable component corresponding to the target operable component information in the user interface of the currently displayed target application changes, a voice prompt message is played to the user indicating that the inventory of the target operable component information has changed.
[0315] For example, the document object model tree of the user interface can be used to determine whether the inventory of the operable component corresponding to the target operable component information in the user interface has changed.
[0316] For example, by comparing screenshots of the user interface at different points in time, it can be determined whether the inventory of the operable component corresponding to the target operable component information in the user interface has changed.
[0317] It is understandable that operable components may be added to the user interface of the target application. For example, if the target application is a beverage application, and a merchant launches a new beverage, the name of the new beverage will be added to the "Beverage Type" field in the "Select Item to Add to Cart" interface. Based on this, this application embodiment also provides the following method: if a new operable component is added to the user interface of the currently displayed target application, a voice prompt will play to inform the user that the new operable component has been added.
[0318] Understandably, some operable components may be removed from the user interface of the target application.
[0319] Understandably, different vehicles have different screen ratios, commonly 4:3, 16:9, and 21:9. The layout of operable components differs across ratios. For example, on a 4:3 screen, the control buttons for the multimedia playback interface might be vertically arranged, while on a 16:9 screen they might be horizontally arranged. If the voice control system tries to operate the user interface displayed on a 16:9 screen based on the position information of the operable components on a 4:3 screen, the operation will fail, meaning it will be difficult to accurately control the corresponding operable components.
[0320] Understandably, applications across different frameworks may follow different rules and conventions when naming operable component information. For example, the operable component information for the "submit" button could be "button-submit," "submit-btn," or "car-submit-button." This naming difference increases the difficulty for voice control systems to recognize and operate operable components in different application scenarios.
[0321] It is understandable that the layout of the user interface of applications with different frameworks may be different or the same. For example, the layout methods that can be used are: flex layout, grid layout, and custom layout. Because the layout of the user interface of applications with different frameworks may be different, the position information and arrangement of the operable components in the user interface will be different.
[0322] It is understandable that the operable component information, location information, and functions of operable components in the target application's user interface may change with version updates and page state changes. Step S204 relies on the accurate identification of operable components to perform control operations. If the above information of operable components is unstable, it is difficult to accurately locate the operable component to be operated and perform the operation. For example, before the update, the operable component information of the operable component "Payment Button" was "Payment". After the update, the operable component information of the operable component "Payment Button" is "Immediate Delivery" or "Pre-sale". If the original operable component information "Payment" is still used, the operable component "Payment Button" cannot be located, thus causing the control operation to fail.
[0323] For example, the functions of operable components at the same location on the same user interface in the target application after the update and before the update may be completely different. For instance, the function of the operable component at location A on the user interface of the target application before the update is to add to cart; while the function of the operable component at location A on the user interface of the target application after the update is to pay immediately.
[0324] In summary, the location information and operable component information of the user interface displayed by the target application may change, and the order of multiple user interfaces displayed by the target application may also change. If the voice control system still operates the updated user interface according to the previous location information and operable component information, the operation may fail, and the path needs to be replanned. Based on this, the first implementation method of step S204 "controlling the operation of the operable component corresponding to the target operable component information in the user interface" provided in this application embodiment specifically includes the following steps H11 to H13.
[0325] Step H11: Obtain the third document object model tree corresponding to the user interface of the currently displayed target application.
[0326] Step H12: Obtain the target location information and target operation method of the operable component corresponding to the target operable component information from the third document object model tree.
[0327] For example, the target operation methods include, but are not limited to: single click and double click.
[0328] Step H13: Operate the target location information of the user interface of the currently displayed target application in the target operation mode.
[0329] The second implementation method of step S204 "controlling the operation of the operable component corresponding to the target operable component information in the user interface" provided in this application embodiment includes the following steps H21 to H25.
[0330] Step H21: Obtain the second image corresponding to the user interface of the currently displayed target application.
[0331] Step H22: Identify the text corresponding to each operable component in the second image.
[0332] Step H23: Calculate the similarity between the text corresponding to each operable component identified in step H22 and the information of the target operable component.
[0333] Step H24: Determine the location information of the operable component corresponding to the text with the highest similarity calculated in step H23 as the location information of the target operable component.
[0334] Step H25: Operate the target location information of the user interface of the currently displayed target application.
[0335] As can be seen from the first and second implementation methods, this application designs an anchor point adaptive mechanism to detect changes in the operable components corresponding to the target operable components in real time, so as to automatically adjust the operation path.
[0336] It is understandable that both the first and second implementation methods obtain the position and information of the operable components in real time during the process of controlling the operable components corresponding to the target operable component information in the user interface, thus enabling precise operation.
[0337] It is understandable that if the position information and operable component information of the operable components in the user interface are obtained in advance, during the process of "controlling the operable components corresponding to the target operable component information in the user interface", if the position information and operable component information of the operable components in the user interface change, it may cause the operation to fail; if the operation fails, the first implementation method or the second implementation method described above can be executed again.
[0338] In an alternative implementation, the success of the user interface transition can be determined by examining the document object model tree of each user interface.
[0339] pass Figures 5a to 5k as well as Figure 6 As can be seen from the above technical solutions, the voice control method provided in this application embodiment is multimodal interaction + SKU (Stock Keeping Unit) + display of special scenarios.
[0340] Special scenarios refer to situations where specific business logic or user needs need to be addressed under certain conditions. For example, if the inventory of a certain SKU is insufficient, users need to be notified promptly, and their ability to purchase products corresponding to that SKU needs to be restricted.
[0341] This application proposes an operable component localization method based on visual features (such as image recognition) and / or DOM tree dual verification to address dynamic changes in the user interface of the target application, such as dialog windows, deletion of operable components, addition of operable components, changes in the location information and information of operable components. This method ensures that the operation chain can still be accurately executed when the user interface changes.
[0342] It is understandable that the failure to control the operable component corresponding to the target operable component information in the user interface may be due to the following three reasons.
[0343] One possible reason is that the user's voice command missed information about the operable components in the user interface of the currently displayed target application. In this case, a voice prompt can be played to allow the user to input the corresponding voice command.
[0344] Reason 2: Operation on the operable components in the user interface of the currently displayed target application timed out.
[0345] For example, a retry mechanism can be automatically triggered. For example, a retry threshold can be preset. If the retry threshold is exceeded, it is confirmed that the control operation of the operable component corresponding to the target operable component information in the user interface has failed, and a prompt message is played by voice.
[0346] For example, the specific value of the retry threshold can be determined based on the actual situation, such as 3.
[0347] Reason three: The operable components in the user interface of the currently displayed target application have changed. The operable component information corresponding to the voice command includes the operable component information of the currently displayed target application's user interface before the update, but does not include the operable component information of the currently displayed target application's user interface after the update. In this case, a voice prompt can also be played to allow the user to input the corresponding voice command. Alternatively, steps J1 to J4 can be executed.
[0348] Step J1: Obtain the second document object model tree corresponding to the target application.
[0349] For an explanation of the second document object model tree, please refer to the explanation of the first document object model tree; it will not be repeated here.
[0350] Step J2: Obtain page navigation relationship information based on the second document object model tree. The page navigation relationship information includes the navigation relationship of multiple operable components and the position information of the operable components in the user interface.
[0351] For example, page navigation information can be obtained in advance. For example, page navigation information can be updated periodically.
[0352] Step J3: If the control operation of the operable component corresponding to the target operable component information in the user interface fails, obtain the first jump path from the user interface in the currently displayed target application to the target user interface from the page jump relationship information; the target user interface is the user interface located after the currently displayed user interface and closest to the currently displayed user interface and has the operable component corresponding to the target operable information.
[0353] Step J3 achieves the goal of real-time fault-tolerant planning, which is to start the alternative path, i.e. the first jump path, when a loading exception occurs.
[0354] For example, there are several situations in which the control operation of the operable component corresponding to the target operable component information in the user interface fails. The embodiments of this application provide, but are not limited to, the following five situations.
[0355] Scenario 1: The user interface of the target application currently displayed is unrelated to the implementation of the voice commands.
[0356] For example, in the process of implementing the voice command "order an iced Americano," the target application displays multiple user interfaces in the following order: store selection interface → order selection interface → new product recommendation interface → select product and add to cart interface → order checkout interface → beverage preparation progress interface. In implementing the voice command "order an iced Americano," the multiple user interfaces associated with it include: store selection interface, order selection interface, select product and add to cart interface, order checkout interface, and beverage preparation progress interface, but exclude the "new product recommendation interface," meaning the "new product recommendation interface" is unrelated to implementing the voice command "order an iced Americano."
[0357] Scenario 2: The location information and identifier of the operable component corresponding to the target operable component information in the user interface of the currently displayed target application have changed.
[0358] Scenario 3: The updated user interface of the target application has removed the operable components corresponding to the target operable component information.
[0359] Scenario 4: The operable component corresponding to the target operable component information in the user interface of the currently displayed target application is obscured by the dialog window.
[0360] Scenario 5: The operable component corresponding to the target operable component information in the user interface of the currently displayed target application is unavailable.
[0361] Step J4: Operate the user interface in the target application sequentially according to the first jump path to control the display of the target user interface.
[0362] For example, the first redirect path involves multiple user interfaces that do not require the user to input any information, but are simply to redirect to a user interface containing the target.
[0363] For example, the confidence level of the first redirection path can be calculated. If the confidence level is low, a prompt message can be played to ask the user whether they can jump to the user interface through the first redirection path. If the confidence level is high, step J4 can be executed directly.
[0364] For example, the optimal first jump path can be obtained based on a Monte Carlo tree search path optimization method.
[0365] For example, the first redirect path involves the fewest number of user interfaces.
[0366] The following example illustrates this. Before the update, the target application's implementation of the voice command "Order an Iced Americano" involved the following user interface display order: Store selection interface → Pickup method selection interface → Select item and add to cart interface → Order checkout interface → Drink preparation progress interface. After the update, the target application's implementation of the voice command "Order an Iced Americano" involves the following user interface display order: Store selection interface → Pickup method selection interface → New product recommendation interface → Select item and add to cart interface → Order checkout interface → Drink preparation progress interface.
[0367] Understandably, users may not need the new products, so they don't need to input any voice commands for the "New Product Recommendation Interface". Assuming the currently displayed user interface is the "New Product Recommendation Interface", the target user interface with the nearest operable component after and closest to the "New Product Recommendation Interface" is the "Select Product and Add to Cart Interface".
[0368] Therefore, if you can jump directly from the "New Product Recommendation" page to the "Select Product and Add to Cart" page, the first jump path is the path from the "New Product Recommendation" page to the "Select Product and Add to Cart" page.
[0369] For example, when displaying the "New Product Recommendation Interface," a voice prompt can be played to ask the user whether they want to purchase the new product, such as "New product recommendation detected, do you want to continue the original order?" This application designs an anchor point adaptive mechanism to detect changes in the operable components corresponding to the target operable component in real time, so as to automatically adjust the operation path or trigger an exception recovery strategy, such as the voice prompt "New product recommendation detected, do you want to continue the original order?"
[0370] For example, if the "Select Items and Add to Cart" interface has an actionable component "Buy Now," but this "Buy Now" component is unavailable, the next best actionable component is automatically selected, such as the actionable component "Add to Cart." In other words, the next best actionable component is the jump component used to navigate to the user interface with the actionable component "Buy Now." That is, the first jump path includes the actionable component "Add to Cart."
[0371] The embodiments of this application can realize cross-page operation chain arrangement, that is, automatically jump between user interfaces and support abnormal path rollback.
[0372] It is understandable that after executing the control operation on the operable component corresponding to the target operable component information in the user interface, changes can still be made. For example, if the user's initial voice command is "Order an iced Americano," and after some time but before payment, the user issues another voice command, "Change the ordered iced Americano to an iced latte," then a modification is needed. If the previous voice command was "Order an iced Americano," and after some time but before payment, the user issues another voice command, "Add another iced latte," then the operable component information to be modified is "latte," and a modification is needed. Based on this, the embodiments of this application also provide the following steps K1 to K3.
[0373] Step K1: Obtain the voice command containing information about the operable components to be modified.
[0374] For example, the operational component information to be modified can be one or more.
[0375] For example, one can also obtain and modify the method, which is to replace or add.
[0376] For example, if the voice command containing the operable component information to be modified is "Change the ordered iced Americano to a cup of iced latte", then the operable component information to be modified is "latte", the modification method is replacement, and at this time, it is necessary to delete the relevant records of "Americano" in the shopping cart.
[0377] For example, if the previous voice command was "order an iced Americano", and the voice command containing the operable component information to be modified is "add another iced latte", then the operable component information to be modified is "latte", and the modification method is "add". At this time, it is necessary to save the relevant records of "Americano" in the shopping cart.
[0378] Step K2: Based on the page jump relationship information, obtain the second jump path from the currently displayed user interface to the user interface with the information of the operable component to be modified.
[0379] Assuming the currently displayed user interface is the payment interface, and the operable component information to be modified is "Latte", then it is necessary to jump from the "payment interface" to the "select product and add to cart interface".
[0380] Step K3: Operate the user interface of the target application sequentially according to the second jump path to control the display of the user interface containing the information of the operable components to be modified.
[0381] For example, the operation can be controlled to display a user interface containing the modified operable component information based on the modified operable component information.
[0382] For example, in addition to reminding users to provide voice feedback by playing prompts through voice playback in the embodiments of this application, tactile prompts can also be provided, such as prompting users to provide voice feedback by vibrating the steering wheel.
[0383] For example, if the embodiments of this application can be applied to in-vehicle terminals, then the embodiments of this application complete a multi-agent architecture centered on professional intelligent agents, and comprehensively build an in-vehicle professional intelligent agent ecosystem; from the perspective of user value, it is equivalent to obtaining a personal assistant to help users operate everything in the vehicle system, based on all multi-step tasks of the vehicle system (such as ordering coffee on the way to work).
[0384] The above describes a voice control method provided by an embodiment of this application. The following describes an apparatus for performing the above voice control method.
[0385] Please see Figure 7 , Figure 7 This is a schematic diagram of the structure of a voice control device provided in an embodiment of this application. Figure 7 As shown, the voice control device includes:
[0386] The first acquisition module 701 is used to acquire voice commands;
[0387] The second acquisition module 702 is used to acquire the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces.
[0388] The first display module 703 is used to display the user interface of the target application with the program identifier;
[0389] The control operation module 704 is used to control the operation of the operable components corresponding to the target operable component information in each user interface of the displayed target application.
[0390] The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
[0391] In one optional implementation, the second acquisition module includes:
[0392] The first acquisition unit is used to acquire keywords in the voice command;
[0393] The second acquisition unit is used to acquire the target slot type to which the keyword belongs and the target operable component information associated with the keyword in the target slot type from a preset domain knowledge graph. The domain knowledge graph includes multiple slot types and operable component information corresponding to the multiple slot types respectively.
[0394] In one optional implementation, the second acquisition module includes:
[0395] The third acquisition unit is used to acquire the target historical voice command that has the highest similarity to the voice command;
[0396] The lookup unit is used to look up the target historical element set corresponding to the target historical voice command from the correspondence between historical voice commands and historical element sets. The target historical element set includes the program identifier and information on multiple target operable components used to implement the target historical voice command.
[0397] In one alternative implementation, it also includes:
[0398] The first playback module is used to play voice prompts to indicate whether the user's current needs are consistent with the needs of the target historical element set.
[0399] The second display module is used to display a first window containing the target historical element set and / or the target historical voice command if a voice command with a defined representation is obtained.
[0400] In an optional implementation, at least one of the target operable component information consists of attribute values of the operable component, and further includes:
[0401] The first identification module is used to identify the first operable component in the user interface that is an interactive type and has not been operated after controlling the operation of the operable component corresponding to the target operable component information in the user interface.
[0402] The second playback module is used to play voice prompts for the user to input the attribute values of the first operable component.
[0403] In one optional implementation, the first identification module includes:
[0404] The fourth acquisition unit is used to acquire the first document object model tree of the user interface. The first document object model tree includes the component type of each operable component in the user interface and the attribute value of each operable component. The component type includes interactive type and non-interactive type.
[0405] The fifth acquisition unit is used to acquire the first operable component that belongs to the interaction type and has not changed, based on the first document object model tree.
[0406] In one optional implementation, the first identification module includes:
[0407] The sixth acquisition unit is used to acquire a first image containing the user interface of the currently displayed target application;
[0408] The first identification unit is used to identify the first operable component in the first image that is in an inactive state.
[0409] In one alternative implementation, it also includes:
[0410] The third playback module is used to play target input prompt information corresponding to the user interface if the target operable component information corresponding to the voice command does not include operable component information displayed by the user interface. The target input prompt information is a prompt message that prompts the user to input operable component information displayed by the user interface.
[0411] In one alternative implementation, it also includes:
[0412] The third acquisition module is used to acquire information about operable components in the user interface of the currently displayed target application;
[0413] The fourth acquisition module is used to acquire the target input prompt information corresponding to the operable component information in the user interface of the currently displayed target application from a plurality of preset input prompt information.
[0414] In one alternative implementation, it also includes:
[0415] The third display module is a second window used to display the progress information of implementing the voice command in real time. The progress information includes the target input prompt information corresponding to the displayed user interface and the target operable component information corresponding to the target input prompt information. The target input prompt information corresponding to the user interface is a prompt information prompting the user to input the operable component information displayed by the user interface.
[0416] In one alternative implementation, it also includes:
[0417] The fourth display module is used to display the user interface of the target application after the operation of the operable components is completed.
[0418] In one alternative implementation, it also includes:
[0419] The close operation module is used to close dialog windows that obscure the user interface of the currently displayed target application.
[0420] In one optional implementation, the shut-down operation module includes:
[0421] The first determining unit is used to determine that there is a dialog window that obscures the user interface of the currently displayed target application;
[0422] The second identification unit is used to identify the window type of the dialog window;
[0423] The first playback unit is used to play a voice prompt message to the user to authorize if the window type is an authorization type.
[0424] A control operation unit is used to respond to a voice command containing an authorization result and control the operation of the authorization result in the dialog window;
[0425] The first closing operation unit is used to close the dialog window;
[0426] The second closing operation unit is used to close the dialog window if the window type is an unauthorized type.
[0427] In one alternative implementation, it also includes:
[0428] The fourth acquisition module is used to acquire the second document object model tree corresponding to the target application;
[0429] The fifth acquisition module is used to acquire page jump relationship information based on the second document object model tree. The page jump relationship information includes the jump relationship of multiple operable components and the position information of the operable components in the user interface.
[0430] In one alternative implementation, it also includes:
[0431] The sixth acquisition module is used to acquire, from the page jump relationship information, a first jump path from the user interface in the currently displayed target application to the target user interface if the control operation of the target operable component information corresponding to the operable component in the user interface fails; the target user interface is the user interface located after the currently displayed user interface and closest to the currently displayed user interface and has the operable component corresponding to the target operable information.
[0432] The first path operation module is used to sequentially operate the user interface in the target application according to the first jump path in order to control the display of the target user interface.
[0433] In one alternative implementation, it also includes:
[0434] The seventh acquisition module is used to acquire voice commands containing information about the operable components to be modified.
[0435] The eighth acquisition module is used to acquire, based on the page jump relationship information, a second jump path from the currently displayed user interface to the user interface with the information of the operable component to be modified;
[0436] The second path operation module is used to sequentially operate the user interface of the target application according to the second jump path, so as to control the display of the user interface containing the information of the operable components to be modified.
[0437] In one alternative implementation, it also includes:
[0438] The fourth playback module is used to play a voice prompt message to the user indicating that the inventory of the operable component information corresponding to the target operable component information has changed if the inventory of the operable component information corresponding to the target operable component information in the user interface of the currently displayed target application changes.
[0439] In one alternative implementation, it also includes:
[0440] The fifth playback module is used to provide a voice prompt to the user if a new operable component is added to the user interface of the currently displayed target application.
[0441] This application also provides an electronic device in its embodiments. (See reference...) Figure 8The diagram illustrates a structural schematic suitable for implementing the electronic device in the embodiments of this application. The electronic device in the embodiments of this application may include, but is not limited to, fixed terminals such as mobile phones, laptops, PDAs (personal digital assistants), PADs (tablet computers), desktop computers, etc. Figure 8 The electronic device shown is merely an example and should not impose any limitation on the functionality and scope of use of the embodiments of this application.
[0442] like Figure 8 As shown, the electronic device may include a processing unit (e.g., a central processing unit, a graphics processing unit, etc.) 801, which can perform various appropriate actions and processes according to a program stored in a read-only memory (ROM) 802 or a program loaded from a storage device 808 into a random access memory (RAM) 803. When the electronic device is powered on, the RAM 803 also stores various programs and data required for the operation of the electronic device. The processing unit 801, ROM 802, and RAM 803 are interconnected via a bus 804. An input / output (I / O) interface 805 is also connected to the bus 804.
[0443] Typically, the following devices can be connected to I / O interface 805: input devices 806 including, for example, touchscreens, touchpads, keyboards, mice, cameras, microphones, accelerometers, gyroscopes, etc.; output devices 807 including, for example, liquid crystal displays (LCDs), speakers, vibrators, etc.; storage devices 808 including, for example, memory cards, hard drives, etc.; and communication devices 809. Communication device 809 allows electronic devices to communicate wirelessly or wiredly with other devices to exchange data. Although Figure 8 Electronic devices with various devices are shown, but it should be understood that it is not required to implement or have all of the devices shown. More or fewer devices may be implemented or have alternatively.
[0444] This application also provides a computer program product including computer-readable instructions, which, when executed on an electronic device, cause the electronic device to implement any of the voice control methods provided in this application.
[0445] This application also provides a computer-readable storage medium that carries one or more computer programs. When the one or more computer programs are executed by an electronic device, the electronic device can implement any of the voice control methods provided in this application.
[0446] It should also be noted that the device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate, and the components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs. In addition, in the device embodiment drawings provided in this application, the connection relationship between modules indicates that they have a communication connection, which can be implemented as one or more communication buses or signal lines.
[0447] Through the above description of the embodiments, those skilled in the art can clearly understand that this application can be implemented by means of software plus necessary general-purpose hardware, or it can be implemented by special-purpose hardware including application-specific integrated circuits, special-purpose CPUs, special-purpose memory, special-purpose components, etc. Generally, any function performed by a computer program can be easily implemented by corresponding hardware, and the specific hardware structure used to implement the same function can also be diverse, such as analog circuits, digital circuits, or special-purpose circuits. However, for this application, software program implementation is more often the preferred implementation method. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, can be embodied in the form of a software product. This computer software product is stored in a readable storage medium, such as a computer floppy disk, USB flash drive, mobile hard disk, ROM, RAM, magnetic disk, or optical disk, etc., and includes several instructions to cause a computer device (which may be a personal computer, training equipment, or network device, etc.) to execute the methods described in the various embodiments of this application.
[0448] In the above embodiments, implementation can be achieved, in whole or in part, through software, hardware, firmware, or any combination thereof. When implemented in software, it can be implemented, in whole or in part, as a computer program product.
[0449] The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or part of the processes or functions described in the embodiments of this application are generated. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, training device, or data center to another website, computer, training device, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) means. The computer-readable storage medium may be any available medium that a computer can store or a data storage device such as a training device or data center that integrates one or more available media. The available media may be magnetic media (e.g., floppy disks, hard disks, magnetic tapes), optical media (e.g., DVDs), or semiconductor media (e.g., solid-state drives (SSDs)).
Claims
1. A voice control method, characterized in that, include: Get voice commands; Obtain the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces; Display the user interface of the target application with the program identifier; For each user interface of the target application shown, control the operation of the operable component corresponding to the target operable component information in that user interface; The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
2. The voice control method according to claim 1, characterized in that, Obtaining information on multiple target operable components corresponding to the voice command includes: Obtain keywords from the voice command; The target slot type to which the keyword belongs and the target operable component information associated with the keyword in the target slot type are obtained from a preset domain knowledge graph. The domain knowledge graph includes multiple slot types and operable component information corresponding to each of the multiple slot types.
3. The voice control method according to claim 1, characterized in that, The step of obtaining the program identifier corresponding to the voice command and information on multiple target operable components includes: Obtain the target historical voice command that has the highest similarity to the stated voice command; From the correspondence between historical voice commands and historical element sets, find the target historical element set corresponding to the target historical voice command. The target historical element set includes the program identifier and information on multiple target operable components used to implement the target historical voice command.
4. The voice control method according to claim 3, characterized in that, After searching for the target historical element set corresponding to the target historical voice command from the correspondence between historical voice commands and historical element sets, the method further includes: The system plays a voice prompt asking the user whether their current needs match the needs of the target historical element set. If a voice command with a defined representation is obtained, a first window containing the target historical element set and / or the target historical voice command is displayed.
5. The voice control method according to any one of claims 1 to 4, characterized in that, At least one of the target operable component information is an attribute value of the operable component, the voice command has a first voice command including the attribute value of the target operable component, and before obtaining the first voice command, it further includes: After controlling the operation of the operable component corresponding to the target operable component information in the user interface, the first operable component in the user interface that is of the interaction type and has not been operated is identified. The voice prompt is used to prompt the user to input the attribute value of the first operable component.
6. The voice control method according to claim 5, characterized in that, The first operable component in the user interface that is an interaction type and has not been operated includes: Obtain the first document object model tree of the user interface, the first document object model tree including the component type of each operable component in the user interface and the attribute value of each operable component, the component type including interactive type and non-interactive type; The first operable component that belongs to the interaction type and has not changed is obtained based on the first document object model tree.
7. The voice control method according to claim 5, characterized in that, The first operable component in the user interface that is an interaction type and has not been operated includes: Obtain a first image containing the user interface of the target application currently being displayed; Identify the first operable component in the first image that is in an inactive state.
8. The voice control method according to any one of claims 1 to 7, characterized in that, Before controlling the operation of the operable components in the user interface that have the operable component information, the method further includes: If the target operable component information corresponding to the voice command does not include the operable component information displayed by the user interface, the voice plays the target input prompt information corresponding to the user interface. The target input prompt information is a prompt that prompts the user to input the operable component information displayed by the user interface.
9. The voice control method according to claim 8, characterized in that, The process of obtaining the target input prompt information corresponding to the user interface of the currently displayed target application includes: Obtain information about the operable components in the user interface of the currently displayed target application; From a plurality of preset input prompts, obtain the target input prompt information corresponding to the operable component information in the user interface of the currently displayed target application.
10. The voice control method according to any one of claims 1 to 9, characterized in that, Also includes: A second window displays the progress information of implementing the voice command in real time. The progress information includes the target input prompt information corresponding to the displayed user interface and the target operable component information corresponding to the target input prompt information. The target input prompt information corresponding to the user interface is a prompt information prompting the user to input the operable component information displayed by the user interface.
11. The voice control method according to any one of claims 1 to 10, characterized in that, Also includes: The user interface of the target application is displayed after the operation of the operable components is completed.
12. The voice control method according to any one of claims 1 to 11, characterized in that, Also includes: Close the dialog window that obscures the user interface of the target application currently being displayed.
13. The voice control method according to claim 12, characterized in that, The dialog window that closes the user interface of the currently displayed target application includes: Determine if a dialog window exists that is obscuring the user interface of the currently displayed target application; Identify the window type of the dialog window; If the window type is an authorization type, a voice prompt will be played to remind the user to grant authorization. Responding to a voice command containing the authorization result, control the operation of the authorization result in the dialog window; Close the dialog window; If the window type is unauthorized, close the dialog window.
14. The voice control method according to any one of claims 1 to 13, characterized in that, Also includes: Obtain the second document object model tree corresponding to the target application; Page navigation relationship information is obtained based on the second document object model tree. The page navigation relationship information includes the navigation relationship of multiple operable components and the position information of the operable components in the user interface.
15. The voice control method according to claim 14, characterized in that, Also includes: If controlling the operable component corresponding to the target operable component information in the user interface fails, from the page... The jump relationship information retrieves the jump from the currently displayed user interface of the target application to the target user interface. The first side A jump path; the target user interface is the user interface that is located after the currently displayed user interface and is closest to the currently displayed user interface, and has the operable components corresponding to the target operable information; The user interface in the target application is operated sequentially according to the first jump path to control the display of the target user interface.
16. The voice control method according to any one of claims 14 to 15, characterized in that, After controlling the operation of the operable component corresponding to the target operable component information in the user interface, the method further includes: Obtain voice commands containing information about operable components to be modified; Based on the page jump relationship information, obtain a second jump path from the currently displayed user interface to the user interface with the operable component information to be modified; The user interface of the target application is operated sequentially according to the second jump path to control the display of the user interface containing the information of the operable components to be modified.
17. The voice control method according to any one of claims 1 to 16, characterized in that, Also includes: If the inventory of the operable component corresponding to the target operable component information in the currently displayed user interface of the target application changes, a voice prompt will play to inform the user that the inventory of the target operable component information has changed.
18. The voice control method according to any one of claims 1 to 17, characterized in that, Also includes: If a new operable component is added to the user interface of the target application currently being displayed, a voice prompt will play to inform the user that the new operable component has been added.
19. A voice control device, characterized in that, include: The first acquisition module is used to acquire voice commands; The second acquisition module is used to acquire the program identifier corresponding to the voice command and multiple target operable component information, wherein at least two of the target operable component information correspond to operable components located in different user interfaces. The first display module is used to display the user interface of the target application with the program identifier; The control operation module is used to control the operation of the operable components corresponding to the target operable component information in each user interface of the displayed target application. The multiple user interfaces of the target application shown are user interfaces used to implement the voice commands.
20. A computer program product, characterized in that, It includes computer-readable instructions that, when executed on an electronic device, cause the electronic device to implement the voice control method as described in any one of claims 1 to 18.
21. An electronic device, characterized in that, It includes at least one processor and a memory connected to the processor, wherein: The memory is used to store computer programs; The processor is used to execute the computer program to enable the electronic device to implement the voice control method as described in any one of claims 1 to 18.
22. A computer storage medium, characterized in that, The storage medium carries one or more computer programs that, when executed by an electronic device, enable the electronic device to implement the voice control method as described in any one of claims 1 to 18.
