User interface control method and apparatus, medium, and electronic device

Abstract

The present application provides a user interface control method and apparatus, a medium, and an electronic device. The present application is applicable to a monitoring program installed in a vehicle-end system. The method comprises: after a voice instruction is acquired, acquiring, from the voice instruction, operation description information of a target interface object in a user interface of any application process on a touch screen; by means of the operation description information, obtaining, from an operation information set, display information and operation action information of the target interface object mapped to the touch screen; and then, on the basis of the display information and the operation action information, performing an operation on the touch screen, and triggering the target interface object to perform a response corresponding to the operation. The monitoring program runs in the vehicle-end system in the form of a process, and is not coupled to other application processes of the vehicle-end system, thereby completely eliminating the need to intervene with the other application programs during development, enabling completely independent monitoring of the other application processes, improving the survivability of the monitoring program, and implementing the function of allowing every visible object to be converted into a spoken instruction.

Description

User interface control method, device, medium and electronic device Technical Field

[0001] The present application relates to the field of voice interaction technology, and in particular to a method, device, medium and electronic device for controlling a user interface. Background Art

[0002] "See it, say it" means that all visible, operable controls on a page can be operated via voice. Operations refer to interactive actions like clicking and sliding, such as clicking the close button on a page to close it. In vehicles, when passengers want to operate a control on the vehicle's computer interface, they can use the "See it, say it" feature to do so, ensuring driving safety.

[0003] In-car Android systems typically include voice control software. Implementing the "see and speak" feature requires knowing the specific content on a page. The voice control software then interprets the instructions and executes the corresponding actions. Adding "see and speak" functionality to new software in the car system requires hacking into and modifying the software's code, which is inefficient.

[0004] Therefore, the present application provides a method for controlling a user interface to solve the above technical problems.

[0005] Summary of the Invention

[0006] The purpose of this application is to provide a user interface control method, device, medium, and electronic device that can solve at least one of the above-mentioned technical problems. The specific solution is as follows:

[0007] According to a specific embodiment of the present application, in a first aspect, the present application provides a method for controlling a user interface, comprising:

[0008] In response to obtaining a voice instruction for a target interface object in a user interface of any application process on the touch screen, obtaining operation description information of the target interface object from the voice instruction;

[0009] Querying an operation information set based on the operation information of the target interface object to obtain display information and operation action information mapped to the touch screen by the target interface object;

[0010] The touch screen is operated based on the display information and the operation action information, triggering the target interface object to respond with a corresponding operation.

[0011] Optionally, operating the touch screen based on the display information and the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0012] Obtaining a position range of the target interface object mapped within the touch screen based on the display information;

[0013] An operation is performed on a position range within the touch screen based on the operation action information, triggering the target interface object to respond with a corresponding operation.

[0014] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0015] When the operation action information includes click operation information, a click operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding click operation.

[0016] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0017] When the operation action information includes sliding operation information, a sliding operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding sliding operation.

[0018] Optionally, in response to obtaining a voice instruction for a target interface object in a user interface of any application process on the touch screen, before obtaining operation description information of the target interface object from the voice instruction, the method further includes:

[0019] In response to acquiring the change information of the user interface, extracting the operation description information of each interface object and the display information and operation action information mapped to the touch screen by the corresponding interface object from the changed user interface;

[0020] An operation information set is constructed based on the operation description information of each interface object and the display information and operation action information mapped to the touch screen by the corresponding interface object, wherein each data item in the operation information set includes the operation description information of the interface object and the correspondence between the display information and operation action information mapped to the touch screen by the corresponding interface object.

[0021] Optionally, the operation description information includes description feature information and description action information.

[0022] Optionally, the descriptive feature information includes index description information.

[0023] According to a specific embodiment of the present application, in a second aspect, the present application provides a user interface control device, comprising:

[0024] an acquiring unit, configured to, in response to acquiring a voice instruction for a target interface object in a user interface of any application process on the touch screen, acquire operation description information of the target interface object from the voice instruction;

[0025] An obtaining unit, configured to query an operation information set based on the operation information of the target interface object, and obtain display information and operation action information of the target interface object mapped onto the touch screen;

[0026] An operating unit is used to operate the touch screen based on the display information and the operation action information, and trigger the target interface object to respond with a corresponding operation.

[0027] Optionally, operating the touch screen based on the display information and the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0028] Obtaining a position range of the target interface object mapped within the touch screen based on the display information;

[0029] An operation is performed on a position range within the touch screen based on the operation action information, triggering the target interface object to respond with a corresponding operation.

[0030] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0031] When the operation action information includes click operation information, a click operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding click operation.

[0032] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0033] When the operation action information includes sliding operation information, a sliding operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding sliding operation.

[0034] Optionally, the device further includes:

[0035] a changing unit configured to, in response to obtaining a voice command for a target interface object in a user interface of any application process on the touch screen, extract, from the changed user interface, the operation description information of each interface object and the display information and operation action information mapped to the corresponding interface object on the touch screen, before obtaining the operation description information of the target interface object from the voice command;

[0036] A construction unit is used to construct an operation information set based on the operation description information of each interface object and the display information and operation action information mapped to the touch screen by the corresponding interface object, wherein each data item in the operation information set includes the operation description information of the interface object and the correspondence between the display information and operation action information mapped to the touch screen by the corresponding interface object.

[0037] Optionally, the operation description information includes description feature information and description action information.

[0038] Optionally, the descriptive feature information includes index description information.

[0039] According to a specific embodiment of the present application, in a third aspect, the present application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements a method for controlling a user interface as described in any one of the above items.

[0040] According to the specific implementation of the present application, in a fourth aspect, the present application provides an electronic device, comprising: one or more processors; a storage device for storing one or more programs, which, when the one or more programs are executed by the one or more processors, enables the one or more processors to implement the user interface control method as described in any of the above items.

[0041] Compared with the prior art, the above solution of the embodiment of the present application has at least the following beneficial effects:

[0042] The present application provides a method, device, medium and electronic device for controlling a user interface. The present application is applied to a monitoring program installed in a vehicle-side system. After obtaining a voice command, the application obtains the operation description information of the target interface object in the user interface of any application process on the touch screen from the voice command, obtains the display information and operation action information of the target interface object mapped to the touch screen from the operation information set through the operation description information, and then operates the touch screen based on the display information and the operation action information, triggering the target interface object to respond to the corresponding operation. The monitoring program runs in the vehicle-side system in the form of a process, and has no coupling relationship with other application processes of the vehicle-side system. During the development process, there is no need to invade other applications at all, and it can monitor other application processes completely independently, thereby improving the survivability of the monitoring program and realizing the function of seeing and saying. BRIEF DESCRIPTION OF THE DRAWINGS

[0043] FIG1 shows a flow chart of a method for controlling a user interface according to an embodiment of the present application;

[0044] FIG2 shows a unit block diagram of a control device for a user interface according to an embodiment of the present application. DETAILED DESCRIPTION

[0045] To make the objectives, technical solutions, and advantages of this application more clear, this application will be further described in detail below with reference to the accompanying drawings. Obviously, the embodiments described are only some of the embodiments of this application, not all of them. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making any creative efforts are within the scope of protection of this application.

[0046] The terms used in the examples of this application are for the purpose of describing specific embodiments only and are not intended to limit this application. The singular forms "a," "the," and "the" used in the examples of this application and the appended claims are also intended to include plural forms, and unless the context clearly indicates otherwise, "a plurality" generally includes at least two.

[0047] It should be understood that the term "and / or" as used herein is merely a description of the relationship between associated objects, indicating that three possible relationships exist. For example, "A and / or B" can represent: A exists alone, A and B exist simultaneously, or B exists alone. Furthermore, the character " / " in this document generally indicates that the associated objects are in an "or" relationship.

[0048] It should be understood that although the terms first, second, third, etc. may be used to describe in the embodiments of the present application, these descriptions should not be limited to these terms. These terms are only used to distinguish the descriptions. For example, without departing from the scope of the embodiments of the present application, the first may also be referred to as the second, and similarly, the second may also be referred to as the first.

[0049] As used herein, the words "if" and "if" may be interpreted as "at the time of" or "when" or "in response to determining" or "in response to detecting," depending on the context. Similarly, the phrases "if it is determined" or "if (stated condition or event) is detected" may be interpreted as "when it is determined" or "in response to the determination" or "when detecting (stated condition or event)" or "in response to detecting (stated condition or event)," depending on the context.

[0050] It should also be noted that the terms "include," "comprises," or any other variations thereof are intended to encompass non-exclusive inclusion, such that a product or device comprising a series of elements includes not only those elements but also other elements not explicitly listed, or elements inherent to such product or device. In the absence of further limitations, an element defined by the phrase "comprises a..." does not exclude the presence of other identical elements in the product or device comprising the element.

[0051] It should be noted in particular that any symbols and / or numbers in the specification that are not marked in the accompanying drawings are not drawing marks.

[0052] The optional embodiments of the present application are described in detail below with reference to the accompanying drawings.

[0053] The embodiment provided in this application is an embodiment of a method for controlling a user interface.

[0054] The embodiment of the present application is applied to a monitoring program installed in a vehicle-side system. The monitoring program runs in the vehicle-side system in the form of a process and has no coupling relationship with other application processes of the vehicle-side system.

[0055] The embodiment of the present application is described in detail below with reference to FIG1 .

[0056] Step S101 , in response to obtaining a voice instruction for a target interface object in a user interface of any application process on a touch screen, obtaining operation description information of the target interface object from the voice instruction.

[0057] Interface objects are physical elements that can be operated on the user interface, such as controls.

[0058] The target interface object is one of multiple interface objects in the user interface and is also the interface object to be controlled by the voice command.

[0059] Voice instructions include voice description information for operating the target interface object.

[0060] The operation description information includes text information of key information in the voice command.

[0061] Acquiring the operation description information of the target interface object from the voice instruction can be understood as converting the voice instruction into corresponding text information and then extracting the operation description information from the text information.

[0062] Optionally, the operation description information includes description feature information and description action information.

[0063] The descriptive feature information is a description of the features of the target interface object in the user interface. The target interface object in the user interface can be accurately found based on the descriptive feature information.

[0064] Optionally, the descriptive feature information includes index description information.

[0065] Index description information represents the subordinate relationship between key feature words through an index chain. Key feature words at the end of the index chain are subordinate to the key feature words at the beginning. For example, the index description information is <user interface - media player - play button>. Since there is only one media player in the user interface, the media player is subordinate to the user interface; and the media player has only one play button, the play button is subordinate to the player. Using the index description information <user interface - media player - play button>, we can accurately find the media player in the user interface and then find the play button in the media player.

[0066] The descriptive action information is a description of the operation on the target interface object in the user interface. According to the descriptive action information, it is possible to clearly know how to operate the target interface object in the user interface.

[0067] Step S102: querying an operation information set based on the operation description information of the target interface object to obtain display information and operation action information of the target interface object mapped to the touch screen.

[0068] Display information includes the position information, shape information, and size information of the target interface object in the user interface on the touch screen.

[0069] Operation action information refers to the action information required to operate the target interface object on the touch screen, for example, click information on the play button of a media player on the touch screen.

[0070] Each data item in the operation information set includes the operation description information of the interface object and the corresponding relationship between the display information and the operation action information mapped to the touch screen. Since each interface object has at least one operation action, the interface object in the operation information set

[0071] The operation information set is queried based on the operation description information of the target interface object, and the corresponding display information and operation action information can be obtained through the corresponding relationship in the operation information set.

[0072] Step S103: operating the touch screen based on the display information and the operation action information to trigger the target interface object to make a corresponding operation response.

[0073] The embodiment of the present application generates an operation signal for a target interface object by operating the touch screen, so that the target interface object responds to the corresponding operation.

[0074] In some specific embodiments, operating the touch screen based on the display information and the operation action information to trigger the target interface object to respond with a corresponding operation includes the following steps:

[0075] Step S103 - 1 : obtaining a position range of the target interface object mapped on the touch screen based on the display information.

[0076] The display information includes the position information, shape information, and size information of the target interface object in the user interface on the touch screen.

[0077] The position range within the touch screen can be obtained through the position information, shape information and size information in the display information.

[0078] Step S103-2: operate the position range within the touch screen based on the operation action information, and trigger the target interface object to respond with a corresponding operation.

[0079] In some specific embodiments, operating the position range within the touch screen based on the operation action information to trigger the target interface object to respond with the corresponding operation includes the following steps:

[0080] Step S103-2a: When the operation action information includes click operation information, a click operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding click operation.

[0081] For example, in a touch screen, a click operation instruction is generated within the position range of the play button of the corresponding media player, triggering the play button to respond to the corresponding click operation, that is, clicking the play button causes the media player to start playing audio or video.

[0082] In some other specific embodiments, operating the position range within the touch screen based on the operation action information to trigger the target interface object to respond with the corresponding operation includes the following steps:

[0083] Step S103-2b: When the operation action information includes sliding operation information, a sliding operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding sliding operation.

[0084] For example, the target interface object is a user interface. On a touch screen, a sliding operation instruction is generated within the position range corresponding to the user interface, triggering the user interface to respond to the corresponding sliding operation, which is equivalent to performing a sliding operation on the user interface, causing the user interface to change the displayed content.

[0085] In some specific embodiments, the step of obtaining the operation description information of the target interface object from the voice instruction in response to obtaining the voice instruction for the target interface object in the user interface of the touch screen further includes the following steps:

[0086] Step S100 - 1 , in response to obtaining the change information of the user interface, extracting the operation description information of each interface object and the display information and operation action information of the corresponding interface object mapped to the touch screen from the changed user interface.

[0087] Each interface object is extracted from the image of the changed user interface, and the operation description information of the corresponding interface object can be obtained based on the subordinate position relationship of each interface object. The image features of each interface object can be used to determine the operation action of the corresponding interface object, and then the operation action information of the operation action on the touch screen can be obtained. The image position of each interface object can be used to determine the display information of the corresponding interface object on the touch screen.

[0088] Step S100 - 2 : constructing an operation information set based on the operation description information of each interface object and the display information and operation action information of the corresponding interface object mapped to the touch screen.

[0089] Each data item in the operation information set includes operation description information of an interface object and a correspondence between display information and operation action information mapped from the corresponding interface object to the touch screen.

[0090] This specific embodiment provides a method for constructing an operation information set. Whenever the user interface changes, the interface objects in the current user interface also change. Therefore, the operation information set is reconstructed each time so that the data items in the operation information set can be associated with the interface objects in the current user interface and the corresponding voice commands. Therefore, before performing any operation on the interface objects in the user interface, it is necessary to first construct the operation information set.

[0091] The embodiment of the present application is applied to a monitoring program installed in a vehicle-side system. After receiving a voice command, the monitoring program obtains the operation description information of the target interface object in the user interface of any application process on the touch screen from the voice command. The display information and operation action information of the target interface object mapped to the touch screen are obtained from the operation information set through the operation description information. Then, the touch screen is operated based on the display information and the operation action information, triggering the target interface object to respond to the corresponding operation. The monitoring program runs in the vehicle-side system as a process and has no coupling relationship with other application processes of the vehicle-side system. During the development process, there is no need to invade other applications at all. It can monitor other application processes completely independently, improving the viability of the monitoring program and realizing the function of seeing is saying.

[0092] The present application also provides an apparatus embodiment that is based on the above embodiment, and is used to implement the method steps described in the above embodiment. The interpretation based on the same name meaning is the same as that of the above embodiment, and has the same technical effect as the above embodiment, and will not be repeated here.

[0093] As shown in FIG2 , the present application provides a user interface control device 200, comprising:

[0094] An acquiring unit 201 is configured to, in response to acquiring a voice instruction for a target interface object in a user interface of any application process on the touch screen, acquire operation description information of the target interface object from the voice instruction;

[0095] An obtaining unit 202 is configured to query an operation information set based on the operation information of the target interface object, and obtain display information and operation action information of the target interface object mapped onto the touch screen;

[0096] The operating unit 203 is configured to operate the touch screen based on the display information and the operation action information, and trigger the target interface object to respond with a corresponding operation.

[0097] Optionally, operating the touch screen based on the display information and the operation action information to trigger the target interface object to respond with a corresponding operation includes:

[0098] Obtaining a position range of the target interface object mapped within the touch screen based on the display information;

[0099] An operation is performed on a position range within the touch screen based on the operation action information, triggering the target interface object to respond with a corresponding operation.

[0100] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0101] When the operation action information includes click operation information, a click operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding click operation.

[0102] Optionally, operating a position range within the touch screen based on the operation action information to trigger the target interface object to respond to the corresponding operation includes:

[0103] When the operation action information includes sliding operation information, a sliding operation instruction is generated within the position range of the touch screen, triggering the target interface object to respond to the corresponding sliding operation.

[0104] Optionally, the device further includes:

[0105] a changing unit configured to, in response to obtaining a voice command for a target interface object in a user interface of any application process on the touch screen, extract, from the changed user interface, the operation description information of each interface object and the display information and operation action information mapped to the corresponding interface object on the touch screen, before obtaining the operation description information of the target interface object from the voice command;

[0106] A construction unit is used to construct an operation information set based on the operation description information of each interface object and the display information and operation action information mapped to the touch screen by the corresponding interface object, wherein each data item in the operation information set includes the operation description information of the interface object and the correspondence between the display information and operation action information mapped to the touch screen by the corresponding interface object.

[0107] Optionally, the operation description information includes description feature information and description action information.

[0108] Optionally, the descriptive feature information includes index description information.

[0109] The embodiment of the present application is applied to a monitoring program installed in a vehicle-side system. After receiving a voice command, the monitoring program obtains the operation description information of the target interface object in the user interface of any application process on the touch screen from the voice command. The display information and operation action information of the target interface object mapped to the touch screen are obtained from the operation information set through the operation description information. Then, the touch screen is operated based on the display information and the operation action information, triggering the target interface object to respond to the corresponding operation. The monitoring program runs in the vehicle-side system as a process and has no coupling relationship with other application processes of the vehicle-side system. During the development process, there is no need to invade other applications at all. It can monitor other application processes completely independently, improving the viability of the monitoring program and realizing the function of seeing is saying.

[0110] This embodiment provides an electronic device, comprising: at least one processor; and a memory communicatively connected to the at least one processor; wherein the memory stores instructions executable by the one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method steps described in the above embodiment.

[0111] An embodiment of the present application provides a non-volatile computer storage medium, which stores computer-executable instructions. The computer-executable instructions can execute the method steps described in the above embodiment.

[0112] Finally, it should be noted that the various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. References to the common and similar parts between the various embodiments will be sufficient. For the systems or devices disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, their descriptions are relatively simple; for relevant details, refer to the descriptions of the methods.

[0113] The above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit them. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present application.

Claims

1. A control method for a user interface, characterized in that Including: In response to obtaining a voice command for a target interface object in the user interface of any application process on the touch screen, obtaining operation description information of the target interface object from the voice command; Querying an operation information set based on the operation information of the target interface object to obtain display information and operation action information mapped by the target interface object onto the touch screen; Operating on the touch screen based on the display information and the operation action information to trigger a response corresponding to the operation of the target interface object.

2. The method according to claim 1, wherein The operating on the touch screen based on the display information and the operation action information to trigger a response corresponding to the operation of the target interface object includes: Obtaining the position range where the target interface object is mapped within the touch screen based on the display information; Operating on the position range within the touch screen based on the operation action information to trigger a response corresponding to the operation of the target interface object.

3. The method according to claim 2, wherein The operating on the position range within the touch screen based on the operation action information to trigger a response corresponding to the operation of the target interface object includes: When the operation action information includes click operation information, generating a click operation command within the position range on the touch screen to trigger a response corresponding to the click operation of the target interface object.

4. The method according to claim 2, wherein The operating on the position range within the touch screen based on the operation action information to trigger a response corresponding to the operation of the target interface object includes: When the operation action information includes swiping operation information, generating a swiping operation command within the position range on the touch screen to trigger a response corresponding to the swiping operation of the target interface object.

5. The method according to claim 1, wherein Before the step of, in response to obtaining a voice command for a target interface object in the user interface of any application process on the touch screen, obtaining operation description information of the target interface object from the voice command, it further includes: In response to obtaining change information of the user interface, extracting operation description information of each interface object, as well as display information and operation action information mapped by the corresponding interface object onto the touch screen from the changed user interface; Constructing an operation information set based on the operation description information of each interface object and the display information and operation action information mapped by the corresponding interface object onto the touch screen, where each data item in the operation information set includes the corresponding relationship between the operation description information of the interface object and the display information and operation action information mapped by the corresponding interface object onto the touch screen.

6. The method according to claim 1, wherein The operation description information includes description feature information and description action information.

7. The method according to claim 6, wherein The description feature information includes index description information.

8. A control device for a user interface, characterized in that, Including: An obtaining unit, configured to, in response to obtaining a voice command for a target interface object in the user interface of any application process on the touch screen, obtain operation description information of the target interface object from the voice command; An obtaining unit, configured to query an operation information set based on the operation information of the target interface object to obtain display information and operation action information mapped by the target interface object onto the touch screen; An operation unit for operating the touch screen based on the display information and the operation action information, triggering a corresponding operation response from the target interface object.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the program is executed by a processor, it implements the method according to any one of claims 1 to 7.

10. An electronic device, characterized in that, Comprising: One or more processors; A storage device for storing one or more programs, wherein when the one or more programs are executed by the one or more processors, the one or more processors implement the method according to any one of claims 1 to 7.

Images