A voice control method, apparatus and electronic device

Abstract

Embodiments of the present application provide a voice control method, device and electronic equipment. The method is applied to an electronic equipment, and includes: identifying a content element on a current application interface, the content element including an operable control and application interface content other than the operable control; grouping the content element on the current application interface, dividing the operable control and the application interface content other than the operable control having an associated relationship into a group, and obtaining a grouping result; obtaining a voice text corresponding to a user input voice; based on the grouping result, confirming an operable control corresponding to an operation object, wherein the operation object is a content element pointed by the voice text; and generating an operation instruction corresponding to the user input voice for the operable control corresponding to the operation object. According to the method of the embodiments of the present application, on the basis of realizing auxiliary voice control, the appearance of the interface can be prevented from being affected, the degree of complexity of voice control operation can be reduced, and thus the user experience of auxiliary voice control can be improved.

Description

Technical Field

[0001] This application relates to the field of computer technology, and in particular to a voice control method, device and electronic device. Background Technology

[0002] In applications that operate terminal devices, many system applications and third-party applications provide voice operation functions to facilitate user operation. Users can operate the application through voice interaction without manual operation.

[0003] However, for system applications and third-party applications that do not provide voice operation functions, users still need to operate them manually. Therefore, a voice interaction method is needed to add voice operation functions to system applications and third-party applications that do not provide voice operation functions. Summary of the Invention

[0004] To address the problem of how to add voice operation functionality to system applications and third-party applications that do not provide voice operation functionality in the existing technology, this application provides a voice control method, device, and electronic device. This application also provides a computer-readable storage medium.

[0005] The embodiments of this application adopt the following technical solutions: In a first aspect, this application provides a voice control method, which is applied to an electronic device, and the method includes: Identify content elements on the current application interface, including operable controls and application interface content other than the operable controls; Group the content elements on the current application interface, group the operable controls with related relationships with the application interface content other than the operable controls into one group, and obtain the grouping results; Get the speech text corresponding to the user's input speech; Based on the grouping results, the operable controls corresponding to the operation objects are identified, wherein the operation objects are the content elements pointed to by the voice text; For the operable controls corresponding to the operation object, generate operation instructions corresponding to the user's voice input.

[0006] In one implementation of the first aspect above, identifying content elements on the current application interface includes: Obtain the Extensible Markup Language (XML) file of the electronic device during runtime, and parse the XML file to obtain the types and positions of the text controls, icon controls, and image controls of the current application interface.

[0007] In one implementation of the first aspect above, identifying content elements on the current application interface includes: Perform Optical Character Recognition (OCR) on the current application interface to obtain the OCR recognition result.

[0008] In one implementation of the first aspect above, identifying content elements on the current application interface includes: Perform icon recognition on the icons on the current application interface and obtain the icon recognition results.

[0009] In one implementation of the first aspect above, identifying content elements on the current application interface includes: Obtain a screenshot of the current application interface, and perform image recognition on the screenshot using an application interface recognition model to identify operable controls on the current application interface.

[0010] In one implementation of the first aspect above, the step of grouping the content elements on the current application interface and obtaining the grouping results includes: Group the operable controls in the content element with their corresponding text descriptions.

[0011] In one implementation of the first aspect above, the step of grouping the content elements on the current application interface and obtaining the grouping results includes: Group the content elements belonging to the same list item in the current application interface into a group.

[0012] In one implementation of the first aspect above, the step of grouping the content elements on the current application interface and obtaining the grouping results includes: The association relationship of the content elements on the current application interface is detected according to the preset association relationship recognition rules. The association relationship between the content elements is confirmed, and the content elements with the association relationship are grouped together.

[0013] In one implementation of the first aspect described above, the method further includes: Based on the display scheme configuration of the application interface and the control logic of the operable controls, a heuristic relationship recognition rule is designed.

[0014] In one implementation of the first aspect above, the step of grouping the content elements on the current application interface and obtaining the grouping results includes: Based on the control tree in the XML file of the current application interface, determine the relationship between the controls corresponding to each node in the control tree; The controls in the content element are grouped according to the relationships between them.

[0015] In one implementation of the first aspect above, the step of grouping the content elements on the current application interface and obtaining the grouping results includes: The content elements of the current application interface are grouped using a grouping model.

[0016] In one implementation of the first aspect above, the step of confirming the operable control corresponding to the operation object based on the grouping result includes: For the voice text that is a short sentence, identify the operation object that the voice text points to, wherein the operation object is application interface content other than the operable control; Based on the grouping results, the operation object is searched and matched to confirm the operable control corresponding to the operation object.

[0017] In one implementation of the first aspect above, the step of confirming the operable control corresponding to the operation object based on the grouping result includes: For the speech text that is a long sentence, semantic word segmentation is performed on the speech text based on the grouping results to obtain the word segmentation results; The operation object pointed to by the voice text is determined based on the word segmentation result, wherein the operation object is the application interface content other than the operable control; Based on the grouping results, the operation object is searched and matched to confirm the operable control corresponding to the operation object.

[0018] Secondly, this application provides a voice control device, which is applied to an electronic device, and the device includes: The identification module is used to identify content elements on the current application interface, including operable controls and application interface content other than the operable controls. The grouping module is used to group the content elements on the current application interface, grouping the operable controls with relationships with the application interface content other than the operable controls into one group, and obtaining the grouping results. The text acquisition module is used to acquire the voice text corresponding to the user's input voice. The search and matching module is used to identify the operable control corresponding to the operation object based on the grouping results, wherein the operation object is the content element pointed to by the voice text; An operation instruction generation module is used to generate operation instructions corresponding to the user's voice input for the operable controls corresponding to the operation object.

[0019] Thirdly, this application provides an electronic device, which includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the steps of the method described in the first aspect.

[0020] Fourthly, this application provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to perform the method described in the first aspect.

[0021] According to the above-mentioned technical solutions proposed in the embodiments of this application, at least the following technical effects can be achieved: The method according to the embodiments of this application can improve the user experience of assisted voice control by avoiding affecting the aesthetics of the interface and reducing the complexity of voice control operation. Attached Figure Description

[0022] Figure 1 The diagram shown is an application scenario illustration according to an embodiment of this application; Figure 2 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 3 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 4 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 5 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 6 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 7 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 8 The diagram shown is a display interface schematic diagram according to an embodiment of this application; Figure 9 The diagram shown is a flowchart of a method according to an embodiment of this application; Figure 10 The diagram shown is a flowchart of a method execution according to an embodiment of this application; Figure 11 The diagram shown is a simplified structural diagram of a voice operation device according to an embodiment of this application; Figure 12 The diagram shown is a hardware structure schematic of an electronic device according to an embodiment of this application. Detailed Implementation

[0023] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0024] The terminology used in the implementation section of this application is for the purpose of explaining specific embodiments of this application only, and is not intended to limit this application.

[0025] For system applications and third-party applications that do not provide voice operation functionality, a feasible solution to add voice operation capabilities is to provide a voice operation auxiliary application. Specifically, while the system application or third-party application without voice operation functionality is running, the voice operation auxiliary application runs in the background. When the user inputs a voice command, the voice operation auxiliary application recognizes the user's voice input and converts it into a command that can be recognized by the system application or third-party application, thereby enabling voice control of the system application or third-party application.

[0026] Figure 1 The diagram illustrates an application scenario according to an embodiment of this application. The electronic device 100 includes a processor 101, a memory 102, a display device 103, a touch device 104, and a voice input device 105. The memory 102 stores executable program code (e.g., operating system program code and application program code). The processor 101 executes the code stored in the memory 102 to start the operating system and run the corresponding application program. The execution results of the processor 101 can be output to the display device 103. The user can input control commands to the processor 101 by performing touch operations via the touch device 104 according to the content displayed on the display device 103.

[0027] The processor 101 runs the program code corresponding to the first application (which does not support voice operation) on the memory 102, and the display device 103 displays the application interface of the first application. The first application does not support voice operation; the user can only control the first application by performing touch operations through the touch device 104 based on the content displayed on the display device 103, and input control commands to the processor 101.

[0028] To enable voice control for the first application, while the first application is running, the processor 101 executes the program code for the corresponding second application (an application providing voice operation assistance) stored in the memory 102, and the second application runs in the background. The user inputs voice commands via the voice input device 105 based on the content displayed on the display device 103. The background-running second application parses the voice commands and converts them into touch commands that can be recognized by the first application.

[0029] exist Figure 1 In the application scenario shown, the key to achieving voice control lies in the second application needing to convert voice commands into touch commands that can be recognized by the first application. In one feasible implementation, the second application converts the voice command into text, and then uses the text corresponding to the voice command to match the text on the application interface (Graphical User Interface, GUI), thereby converting the voice command into a touch command that can be recognized by the first application.

[0030] Specifically, after the second application receives the user's voice command, it converts the voice command into text, parses the text corresponding to the voice command to determine the first text corresponding to the operation method and the second text corresponding to the operation target. Furthermore, the second application parses the text on the currently displayed interface to identify a third text on the currently displayed interface that matches the second text. The second application generates a touch command whose operation target is the third text, and whose operation method corresponds to the first text. The second application sends the generated touch command to the first application, thereby enabling voice control of the first application.

[0031] For example, Figure 2 The diagram shown is a display interface schematic according to an embodiment of this application. In an application scenario, the running interface of the first application is as follows: Figure 2 As shown. Figure 2 To display the interface diagram, the text "First Column / Second Column / Third Column / Fourth Column / Fifth Column", "First Special Zone / Second Special Zone", and "Album 1 / Album 2 / Album 3 / Album 4 / Album 5" can be any text.

[0032] The first application does not have voice interaction capabilities. To provide voice operation functionality for this music application, a voice operation assistance application runs in the background while the music application is running. When the user inputs "click on the fifth column" via voice, the voice operation assistance application recognizes the voice "click on the fifth column", converts the voice "click on the fifth column" into text, and obtains the corresponding operation "click" and the operation object "fifth column".

[0033] Furthermore, voice-assisted operation applications can recognize... Figure 2The text on the displayed interface is identified as "Fifth Column" (the text within the dotted box 201), which corresponds to the text "Fifth Column" on the displayed interface. The operation "click" is then performed on the text "Fifth Column" (the text within the dotted box 201), thus ultimately converting the user's voice input "click the fifth column" into a click. Figure 2 The click operation of the text "Fifth Column (text in dashed box 201)" on the displayed interface.

[0034] While the above implementation method can achieve voice control of operable text controls on the application interface, it relies on the presence of corresponding text at the location of the operable control. However, in real-world applications, to optimize the user's visual experience and responsiveness, various display methods for operable controls exist on the application interface. This makes it possible for operable controls on the application interface to be uncontrollable using the above implementation method. The following three specific application scenarios (application scenarios 1.1, 1.2, 1.3, and 1.4) illustrate how the display method of operable controls on the application interface can prevent voice control based on the above implementation method.

[0035] Application Scenario 1.1 exist Figure 2 On the application interface shown, images 211, 212, 213, 214, and 215 are the covers of different music albums. Clicking on one of the images 211, 212, 213, 214, or 215 will take you to the song list of the corresponding music album. The text markers corresponding to images 211, 212, 213, 214, and 215 are located below the images (the text within the dotted boxes 221, 222, 223, 224, and 225).

[0036] If a user wants to use voice control to click on image 211, the user voice-inputs "Click Album 1". The voice-assisted operation application recognizes the voice "Click Album 1", converts it into text, and obtains the corresponding operation "Click" and the operation object "Album 1". The voice-assisted operation application recognizes... Figure 2 The text on the displayed interface is "Album 1 (dashed box 221)". The operation to "click" is performed on the text "Album 1 (dashed box 221)". However, the text "Album 1 (dashed box 221)" on the application interface is not an operable control; its corresponding operable control is the image 211 above the text. Therefore, the user cannot use voice control to click on image 211.

[0037] Application Scenario 1.2 Figure 3The diagram shown is a display interface schematic according to an embodiment of this application. In one application scenario, after entering the system's desktop settings page, the display interface is as follows. Figure 3 As shown.

[0038] If a user wants to use voice control to turn on the "Lock Layout" switch, the user voice-activated input "Click to lock layout." The voice-assisted application recognizes the voice input "Click to lock layout," converts it into text, and obtains the corresponding action "click" and the target "lock layout." The voice-assisted application then recognizes the input. Figure 3 The text on the displayed interface is "Lock Layout (text in dashed box 301)". The user then performs a "click" operation on the text "Lock Layout (text in dashed box 301)". However, the text "Lock Layout (text in dashed box 301)" on the application interface is not an operable control; its corresponding operable control (switch 311) is located to the right of the text. Therefore, the user cannot use voice control to turn on the "Lock Layout" switch.

[0039] Application Scenario 1.3 Figure 4 The diagram shown is a display interface schematic according to an embodiment of this application. In an application scenario, such as Figure 4 As shown, the comment browsing page displays the name of the user who posted the comment (e.g., the user name "User A" in dashed box 401), the specific comment content (e.g., the comment content in dashed box 411), and other users' replies to the comment (e.g., User B's reply in dashed box 421).

[0040] Figure 4 The diagram shown is a schematic diagram of the display interface, and is an example of the interface schematic diagram of an embodiment in this application specification. Figure 4 as well as Figure 4 In the subsequent interface diagrams, “XXXXXX” can be any text.

[0041] Furthermore, for each comment, below the comment content or comment reply, there are buttons for "forward", "reply", and "like" (for example, button icon 431 for forwarding, button icon 441 for replying, and button icon 451 for liking). Users can click the buttons to perform the "forward", "reply", and "like" operations for the corresponding comment content.

[0042] Suppose a user wants to use voice control to click icon 451 (to like the comment content within the dotted box 411). The user voice-activated input says "Click the like button." The voice-activated application recognizes the voice "Click the like button," converts it into text, and obtains the corresponding action "click" and the action object "like button." The voice-activated application recognizes... Figure 4 The text shown on the display interface, however, is due to Figure 4 The displayed interface shows that the like button is only an icon and does not contain any text. Therefore, users cannot use voice control to click the like button.

[0043] To address the problems in the aforementioned voice control implementation methods, a feasible solution is to perform semantic recognition on all operable controls on the display interface (based on Optical Character Recognition (OCR) text or icon recognition methods), recognize the semantics of operable controls (e.g., buttons, icons, images), and use the text of the voice command to match the text on the application interface, thereby converting the voice command into a touch command that can be recognized by the application.

[0044] For example, targeting Figure 4 The application interface shown illustrates this: if a user voice-inputs "click the like button," the voice-assisted application recognizes the voice input, converts it into text, and retrieves the corresponding action "click" and the action object "like button." The voice-assisted application then recognizes the text. Figure 4 The operable controls on the display interface shown are based on an icon recognition method. The semantics of icon 451 are recognized as "like", thus confirming that icon 451 is the operation object "like button". The user's voice input "click like button" is then converted into a click operation of icon 451.

[0045] While the above solution can convert voice input by recognizing operable controls on the display interface, it is not universally applicable to all application interfaces. For example, on some application interfaces, there are multiple operable controls with the same semantic meaning. When the user's voice operation target is related to the semantic meaning of these operable controls, the voice operation assistance application cannot determine which of these operable controls is being operated on.

[0046] For example, targeting Figure 2In the application interface shown, the semantics of the operable controls (images 211, 212, 213, 214, 215) can all be recognized as "album cover images". When the user inputs the voice command "click on album 1 cover image", the voice operation assistance application can map the operation object of "click on album 1 cover image" to the operable controls (images 211, 212, 213, 214, 215) with the semantics of "album cover image". However, the voice operation assistance application cannot confirm which of images 211, 212, 213, 214, 215 corresponds to "album 1 cover image".

[0047] For example, regarding Figure 3 In the application interface shown, the semantics of the operable control switch 311 corresponding to "Lock Layout" and the operable control switch 312 corresponding to "Auto Align" can both be recognized as "switches". If the user voice inputs "Click the lock layout switch", the voice operation assistance application recognizes the voice "Click the lock layout switch" and identifies... Figure 3 The operable controls on the displayed interface, however, are... Figure 3 On the displayed interface, both switches 311 and 312 have the semantic meaning of "switch". The voice operation assistance application cannot confirm which of switches 311 and 312 is being operated on when "click to lock layout switch".

[0048] For example, Figure 5 The diagram shown is a display interface schematic according to an embodiment of this application. In an application scenario, such as Figure 5 As shown, the browsing page for comments displays three comments. For each comment, the names of the users who posted the comments are displayed: User B (text in dashed box 541), User C (text in dashed box 542), and User A (text in dashed box 543); the specific comment content is displayed as follows: comment text in dashed box 501, comment text in dashed box 502, and comment text in dashed box 503; and the replies from other users to the comment are displayed as follows: reply text from User D in dashed box 551 and reply text from User E in dashed box 552.

[0049] Furthermore, for each comment, there are icon buttons below the comment content or comment reply for "forward", "reply", and "like". For example, icons 511, 512, and 513 are for forwarding comments, icons 521, 522, and 523 are for replying, and icons 531, 532, and 533 are for liking.

[0050] If a user wants to like User B's comment (the comment text in the dashed box 501), the user voice-inputs "like User B." The voice-assisted application recognizes the speech, converts it into text, and confirms that the corresponding action is "click User B's like button." The voice-assisted application then recognizes the text. Figure 5 The operable controls on the displayed interface, however, are... Figure 5 The icons 531, 532, and 533 on the displayed interface all have the meaning of "like button". The voice operation assistance application cannot confirm which of the icons 531, 532, and 533 is the target of the "click like button" operation.

[0051] To address the problem of being unable to determine the target of an operation from multiple operable controls with the same semantic meaning, a feasible implementation scheme employs marking the operable controls. When the voice-assisted operation application matches multiple operable controls with the same semantic meaning on the display interface based on the user's voice input, different visual markers are added to these controls. The user then uses these markers to perform further voice input confirmation, thereby determining the final target of the operation.

[0052] For example, Figure 6 The diagram shown is a display interface schematic according to an embodiment of this application. (Regarding...) Figure 2 As shown in the application interface, when the user inputs the voice command "Click on the album 1 cover image," the voice operation assistance application determines the possible target images 211, 212, 213, 214, and 215 based on this command. The voice operation assistance application then adds numerical markers (e.g., ...) to images 211, 212, 213, 214, and 215. Figure 6 (As shown). The user inputs "Image 1" again based on the number marker, and the voice operation assistance application generates the final touch operation command "Click image 211" based on "Image 1".

[0053] For example, Figure 7 The diagram shown is a display interface schematic according to an embodiment of this application. (Regarding...) Figure 3 As shown in the application interface, if the user voice inputs "Click to lock layout switch", the voice operation assistance application will recognize the voice input "Click to lock layout switch" and identify it. Figure 3 The operable controls on the displayed interface, however, are... Figure 3 On the displayed interface, switches 311 and 312 both have the semantic meaning of "switch". The voice operation assistance application adds numerical labels to switches 311 and 312 (e.g., ...). Figure 7 (As shown). The user inputs "Switch 1" again based on the numerical markings, and the voice operation assistance application generates the final touch operation command "Click switch 311" based on "Switch 1".

[0054] For example, Figure 8 The diagram shown is a display interface schematic according to an embodiment of this application. (Regarding...) Figure 5 The application interface shown demonstrates that if a user voice inputs "like for user B", the voice-assisted application will recognize the voice input "like for user B" and then... Figure 5 The operable controls on the displayed interface are shown. Because... Figure 5 The icons 531, 532, and 533 on the displayed interface all signify a "like" button. The voice-assisted operation application adds numerical markers (e.g., ...) to icons 531, 532, and 533. Figure 8 (As shown). The user inputs "Like #1" again based on the numerical marker, and the voice operation assistance application generates the final touch operation command "Click icon 531" based on "Like #1".

[0055] While the aforementioned solutions can distinguish multiple operable controls with the same semantic meaning, they require the voice-assisted application to add visual markers to the operable controls and the user to make a secondary selection. This not only affects the aesthetics of the interface but also significantly increases the complexity of operation, reducing the user experience. Therefore, to simplify the user's voice control process and improve the user experience of voice control, this application proposes a voice control method. In this method, the content elements on the application interface are grouped, and operable controls with related relationships are grouped with other application interface content. In this way, operable controls can be distinguished according to their group, so that when the user's voice command cannot be directly mapped to an operable control on the application interface, the user's voice command can be mapped to an operable control on the application interface according to the grouping relationship of the content elements on the application interface. According to the method of this application embodiment, while achieving assisted voice control, it avoids affecting the aesthetics of the interface, reduces the complexity of voice control operation, and thus improves the user experience of assisted voice control.

[0056] Figure 9 The diagram shows a flowchart of a method according to an embodiment of this application. In one embodiment, the processor 101 executes program code on the memory 102 corresponding to a first application (which does not support voice operation function), and the display device 103 displays the application interface of the first application.

[0057] When the first application runs, the processor 101 executes the program code on the memory 102 corresponding to the second application (an application for providing voice operation assistance). The second application runs in the background. During the execution of the second application, the electronic device 100 executes, for example... Figure 9 The following method flow is shown.

[0058] S900 performs content recognition on the current application interface, recognizing content elements on the current application interface, including operable controls on the current application interface and application interface content other than operable controls.

[0059] The S900 can be implemented in any one or a combination of the following implementation methods 2.1-2.3. Implementation method 2.1 Obtain the Extensible Markup Language (XML) file (e.g., runtime XML file) corresponding to the current application interface in the electronic device 100, and parse the XML file to obtain the type and position of the text control, icon control, and image control of the current application interface (save in the form of a structure).

[0060] Specifically, those skilled in the art can design methods for obtaining XML files based on actual circumstances.

[0061] For example, based on the adb command response of the mobile phone's EMUI system, the runtime XML file of the current interface is obtained at runtime using the adb dumper command.

[0062] Implementation method 2.2 Perform OCR recognition on the current application interface to obtain text recognition results (e.g., text content and / or text box position).

[0063] Specifically, those skilled in the art can design the implementation process of OCR recognition according to the actual situation.

[0064] For example, the system's screenshot function is invoked to capture a screenshot image data stream of the interface (GUI interface). Using the screenshot image as input, the system or cloud-based OCR recognition module is invoked to recognize the paragraph text on the screenshot image. Furthermore, in conjunction with implementation method 2.1, the text recognition result from OCR is merged and corrected with the control resource information extracted from the XML file to obtain the coordinates of the text box corresponding to the text content.

[0065] For example, to obtain Figure 5 The XML file shown is the runtime interface of the application. It utilizes the phone's system screenshot capability to capture images such as... Figure 5 The displayed interface is screenshotted to obtain the screenshot image. Using the screenshot image as input, the phone's system OCR capability is invoked to recognize the text within the screenshot image. For example, the text within the dashed boxes 541 and 542 is recognized. The OCR-recognized text result is merged and corrected with the control resource information extracted from the XML file to obtain the coordinates of the text boxes corresponding to the text content. The final OCR recognition result is shown below.

[0066] Regarding the text within the dashed box 541: {type:text (type: text) Text: "User C" Bound:[100,100,150,120]} (Text box coordinates (top left, bottom right)) Regarding the text within the dashed box 542: {type:text Text: "User B" Bound:[100,500,150,520]} Implementation method 2.3 Perform icon recognition on the icons on the current application interface and obtain the icon recognition results (e.g., icon meaning and / or icon location).

[0067] Specifically, those skilled in the art can design the implementation process of icon recognition according to the actual situation.

[0068] For example, the system's screenshot function is invoked to capture a screenshot image data stream of the interface (GUI). Using the screenshot image as input, the system or cloud-based icon recognition module is invoked to identify the icons on the screenshot image. Furthermore, in conjunction with implementation method 2.1, the icon recognition result is merged and corrected with the control resource information extracted from the XML file to obtain the icon coordinates corresponding to the icon.

[0069] For example, calling the phone's system screenshot capability, such as... Figure 5 The displayed interface is screenshotted to obtain the screenshot image. Using the screenshot image as input, the cloud-based icon recognition module is invoked to identify the icons in the screenshot image. For example, icons 531 and 532 are recognized, and the result indicates that the icons mean "like". The icon recognition result is merged and corrected with the control resource information extracted from the XML file to obtain the icon coordinates. The final icon recognition result is shown below.

[0070] Regarding icon 531: {type:icon (Type: icon)} intent: like (icon intent: like) Bound:[300,100,350,120]} (Icon box coordinates (top left, bottom right)) For icon 532: {type: icon intent: like Bound:[300,500,350,520]} Implementation method 2.4 Get a screenshot of the current application interface, and use the application interface recognition model to perform image recognition on the screenshot to identify the operable controls on the current application interface.

[0071] Specifically, screenshots of application interfaces from different applications in different operating scenarios are collected, and the buttons, images, and text on the screenshots are labeled to generate learning model samples. The application interface recognition model is then trained based on the learning model samples.

[0072] S910: Group the content elements on the current application interface, grouping the operable controls with relationships with other application interface content, and obtain the grouping results.

[0073] In this way, operable controls can be distinguished according to the group to which they belong. Thus, when a user's voice command cannot be directly mapped to an operable control on the application interface, the user's voice command can be mapped to an operable control on the application interface based on the grouping relationship of content elements on the application interface.

[0074] The implementation logic of S910 includes at least one or a combination of two of the following grouping logics 1-2.

[0075] Grouping Logic 1 The operable controls and their corresponding text descriptions are grouped together. Specifically, in S910, content elements on the application interface (e.g., buttons, icons, images, and text) are identified, and the relationships between text + icon, text + image, and container text + icon on the application interface are analyzed. Related text + icon, text + image, and container text + icon combinations are grouped together. For example, an image and its description text are grouped together; an icon and its description text are grouped together.

[0076] For example, targeting Figure 2 In the application interface shown, images 211-215 are associated with the text within the dashed boxes 221-225 below them (the text explains the content of the images). Therefore, image 211 is grouped with the text in dashed box 221, image 212 with the text in dashed box 222, image 213 with the text in dashed box 223, image 214 with the text in dashed box 224, and image 215 with the text in dashed box 225. This way, when voice commands are converted to touch commands, operations on the text within dashed boxes 221-225 can be mapped to images 211-215 respectively.

[0077] For example, regarding Figure 3In the desktop settings page shown, switches 311 and 312 are associated with text in dashed boxes 301 and 302, respectively (the text explains the function of the switches). Therefore, switch 311 is grouped with the text "Lock Layout" in dashed box 301; switch 312 is grouped with the text "Auto Align" in dashed box 302. This way, when voice commands are converted to touch commands, the operation targeting the text "Lock Layout" in dashed box 301 can be mapped to switch 311; and the operation targeting the text "Auto Align" in dashed box 302 can be mapped to switch 312.

[0078] Grouping Logic 2 In the list layout on the application interface, content elements belonging to the same list item are grouped together. Specifically, in S910, content elements on the application interface (e.g., buttons, icons, images, and text) are identified, and content elements corresponding to the same list item (content elements belonging to the same list item in the list content) are grouped together.

[0079] For example, such as Figure 5 As shown, the icon 511 for forwarding a comment, the icon 521 for replying, and the icon 531 for liking all correspond to the comment text in the dashed box 501. Furthermore, the username of the user who posted the comment text in the dashed box 501 is User B (the text in the dashed box 541), and the user's reply to the comment text in the dashed box 501 is the reply text in the dashed box 551. Therefore, icons 511, 521, 531, the comment text in the dashed box 501, the text in the dashed box 541, and the reply text in the dashed box 551 are grouped together.

[0080] The icons 512 for forwarding comments, 522 for replying, and 532 for liking all correspond to the comment text in dashed box 502. Furthermore, the username of the user who posted the comment text in dashed box 502 is User C (the text in dashed box 542), and the user's reply to the comment text in dashed box 502 is the reply text in dashed box 552. Therefore, icons 512, 522, 532, the comment text in dashed box 502, the text in dashed box 542, and the reply text in dashed box 552 are grouped together.

[0081] The icons 513 for forwarding comments, 523 for replying, and 533 for liking all correspond to the comment text in dashed box 503. Furthermore, the username of the user who posted the comment text in dashed box 503 is User A (the text in dashed box 543). Therefore, icons 513, 523, 533, the comment text in dashed box 503, and the text in dashed box 543 are grouped together.

[0082] If a user wants to like the comment text in dashed box 501, the user can voice-input "like for user B". The voice assistant application recognizes the speech, converts it into text, and confirms that the corresponding action is "click the like button for user B". Since icons 511, 521, 531, dashed box 501, "user B" (the text in dashed box 541), and the reply text in dashed box 551 are grouped together, when converting the voice command to a touch command, the action "click the like button for user B" can be mapped to icon 511.

[0083] Furthermore, S910 can employ various implementation methods to implement the aforementioned grouping logic. Specifically, S910 can adopt one or any combination of the following implementation methods 3.1-3.3.

[0084] Implementation method 3.1 Based on the application interface display scheme configuration and the control logic of operable controls, a heuristic relationship identification rule is designed. This rule is used to detect relationships between content elements on the current application interface, confirming the relationships between them and grouping related content elements together.

[0085] For example, the positional relationships between images and their descriptions, icons and their descriptions, and switches and their descriptions are determined based on the application interface's display scheme configuration. Based on these positional relationships, and according to the coordinates of the content elements on the current application interface, related content elements are identified and grouped together.

[0086] by Figure 2 Taking the application interface shown as an example, based on the position coordinates of images 211, 212, 213, 214, and 215, and the position coordinates of dashed boxes 221, 222, 223, 224, and 225, it can be determined that dashed boxes 221, 222, 223, 224, and 225 are located immediately below images 211, 212, 213, 214, and 215, respectively. Therefore, it can be determined that the text within dashed boxes 221, 222, 223, 224, and 225 are the descriptive text for images 211, 212, 213, 214, and 215, respectively.

[0087] by Figure 3Taking the application interface shown as an example, based on the position coordinates of switches 311 and 312 and dashed boxes 301 and 302, it can be determined that switch 311 and dashed box 301 are located on the left and right sides of the same row on the screen, and switch 312 and dashed box 302 are also located on the left and right sides of the same row on the screen. Switches 311 and 312 are located in the same column on the screen, and there is empty space between the rows where dashed boxes 301 and 302 are located. Therefore, it can be determined that the text in dashed boxes 301 and 302 are the descriptive text of switches 311 and 312, respectively.

[0088] For example, the page list layout rules of the application interface are determined according to the display scheme configuration of the application interface. Based on the page list layout rules, the list item to which the content element belongs is determined according to the position coordinates of the content element on the current application interface, and content elements belonging to the same list item are grouped together.

[0089] by Figure 5 Taking the application interface shown as an example, according to the page list layout rules, multiple comments are displayed in a list format, with each comment corresponding to one item in the list.

[0090] Based on the position coordinates of icons 511, 512, and 513, it can be determined that icons 511, 512, and 513 are located in the same column on the screen. Since icons 511, 512, and 513 have the same icon, it can be determined that icons 511, 512, and 513 belong to three different list items in the list layout of the current application interface.

[0091] Similarly, icons 521, 522, and 523 belong to three different list items in the current application interface's list layout; icons 531, 532, and 533 belong to three different list items in the current application interface's list layout; the comment text in dashed boxes 501, 502, and 503 belongs to three different list items in the current application interface's list layout; the text in dashed boxes 541, 542, and 543 belongs to three different list items in the current application interface's list layout; and the reply text in dashed boxes 551 and 552 belongs to two different list items in the current application interface's list layout.

[0092] Furthermore, it can be confirmed that the comment text in icons 511, 521, 531, the text in dashed box 501, the text in dashed box 541, and the reply text in dashed box 551 belong to a list item in the list layout of the current application interface; the comment text in icons 512, 522, 532, 502, 542, and 552 belong to a list item in the list layout of the current application interface; and the comment text in icons 513, 523, 533, 503, and 543 belong to a list item in the list layout of the current application interface.

[0093] Implementation method 3.2 Based on the control tree in the XML file of the current application interface, determine the relationships between controls corresponding to each node in the control tree. Group the controls according to the relationships between them.

[0094] For example, based on the obtained XML control tree structure information, the subtrees are segmented and marked to obtain the relationships between icon text, image text, switch controls, and text. Preferably, in conjunction with implementation method 3.1, the results of subtree segmentation and marking are combined with the rules for grouping based on control coordinates to obtain the relationships between icon text, image text, switch controls, and text.

[0095] For example, based on the obtained XML control tree structure information, logical hierarchical relationships are grouped. Specifically, based on the page list layout rules of the application interface, the control corresponding to each list item in the page list is determined. Using the control corresponding to the list item as the terminal node, the runtime XML file of the current application interface is parsed. Tracing back from the leaf node to the next higher level node until the terminal node is reached, all leaf nodes under the same terminal node are grouped together (including the terminal node), thereby obtaining the association relationships between the page list and logical groups on the current application interface.

[0096] Let's take an application scenario as an example. On a bookstore's product display page, different books are displayed in a list format, with each book corresponding to a list item. Each list item includes one or more of the book's title, author, publication date, price, and language. Part of the application's XML code is shown below: <bookstore>(bookstore) <book id="1">(Book number) <name> Book 1< / name> (Book title) <author> Author 1< / author> (Author's name) <year> 2014< / year> (Publication date) <price> 89< / price> (price) < / book> <book id="2">(Book number) <name> Book 2< / name> (Book title) <year> 2004< / year> (Publication date) <price> 77< / price> (price) <language> English< / language> (Language) < / book> <book id="3">(Book number) <name> Book 3< / name> (Book title) <author> Author 3< / author> (Author's name) <year> 2018< / year> (Publication date) <price> 97< / price> (price) <language> German< / language> (Language) < / book> < / bookstore> Using the book ID as the final node for backtracking, the leaf nodes "Book1 (Book Title)", "Author1 (Author Name)", "2014 (Publication Date)" and "89 (Price)" can all be traced back to... <book id="1">(Book number). Therefore, <book id="1">The following are grouped together: "Book ID", "Book 1 (Book Title)", "Author 1 (Author Name)", "2014 (Publication Date)" and "89 (Price)".

[0097] The leaf nodes "Book 2 (Book Title)", "2004 (Publication Date)", "77 (Price)" and "English (Language)" can all be traced back to <book id="2">(Book number). Therefore, <book id="2">The following are grouped together: "Book ID", "Book 2 (Book Title)", "2004 (Publication Date)", "77 (Price)" and "English (Language)".

[0098] The leaf nodes "Book 3 (Book Title)", "Author 3 (Author Name)", "2018 (Publication Date)", "97 (Price)" and "German (Language)" can all be traced back to <book id="3">(Book number). Therefore, <book id="3">The following are grouped together: "Book ID", "Book 3 (Book Title)", "Author 3 (Author Name)", "2018 (Publication Date)", "97 (Price)" and "German (Language)".

[0099] Implementation method 3.3 The content elements of the current application interface are grouped using a grouping model.

[0100] Specifically, the application interfaces of different applications under different operating scenarios are collected, and the content elements in the application interfaces are grouped and labeled to generate learning model samples. The grouped models are then trained based on the learning model samples.

[0101] Furthermore, in practical application scenarios, those skilled in the art can set the triggering conditions for S900 and S910 according to actual application requirements.

[0102] For example, after the second application starts running, upon receiving the user's voice input, S900 and S910 are executed. S900 and S910 are executed every time the user inputs voice data.

[0103] For example, after the second application starts running and collects the user's voice input, S900 is triggered. Based on the recognition result of S900, it is determined whether the text corresponding to the voice input can directly point to an operable control. If so, an operation command corresponding to the voice input is generated based on the recognition result of S900 and the text corresponding to the voice input. If the text corresponding to the voice input cannot directly point to an operable control, S910 is triggered.

[0104] For example, for a specific application interface, S900 and S910 are only triggered when the user makes their first voice input. In S910, the grouping results of the content elements of the current application interface are saved. While the current application interface remains unchanged, S900 and S910 are not triggered again.

[0105] For example, after the second application starts running, when the application interface of the first application changes, S900 is executed for the latest application interface (the current application interface); in S910, the grouping results of the content elements of the current application interface are saved. While the current application interface remains unchanged, the execution of S900 and S910 is not triggered again.

[0106] For example, after the second application starts running, after collecting the user's voice input, the execution of S900 is triggered. Based on the recognition result of S900, it is confirmed whether the grouping result for the current application interface has been saved. If the grouping result for the current application interface has been saved, it is directly called; if the grouping result for the current application interface has not been saved, the execution of S910 is triggered. In S910, the grouping result of the content elements of the current application interface is saved.

[0107] Furthermore, in practical application scenarios, those skilled in the art can set the method for saving the grouping results of S910 according to actual application needs.

[0108] For example, in one implementation, an association matrix is ​​used to store the grouping results of S910. Each row in the association matrix corresponds to a content element on the application interface, and each column also corresponds to a content element on the application interface. The value (0 or 1) of a matrix element in the association matrix represents the association between the content element corresponding to the row of that matrix element and the content element corresponding to the column of that matrix element (0 indicates no association, 1 indicates association).

[0109] S920, obtains the voice text corresponding to the user's input voice.

[0110] Specifically, prior to S920, electronic device 100 collected user voice input and converted it into speech text based on Automatic Speech Recognition (ASR).

[0111] S921 performs semantic matching on the speech text to confirm whether the speech text is an executable short sentence that can be directly matched to the current application interface.

[0112] When the voice text is an executable short phrase that can be directly matched to the current application interface, execute S930.

[0113] S930: Based on the voice text, confirm the target of the operation, which is application interface content other than the operable controls on the current application interface.

[0114] For example, targeting Figure 3 As shown in the interface, when the user inputs "Lock Layout," the voice text can be directly matched with the text "Lock Layout" in the dashed box 301. The voice text is a short, directly executable phrase; therefore, the object the voice text points to is "Lock Layout." The text "Lock Layout" in the dashed box 301 is not an operable control; therefore, the object the voice text points to is application interface content outside of the operable controls on the current application interface.

[0115] S931, based on the grouping results of S910, searches and matches the operation object pointed to by the voice text, and confirms the operable control corresponding to the operation object pointed to by the voice text.

[0116] For example, targeting Figure 3 The interface shown allows you to directly search and match the voice text "Lock Layout" to confirm the operable control (switch 311) that the voice text "Lock Layout" points to.

[0117] In S930, if the object of operation is an operable control on the current application interface, then S931 can be skipped and the operable control corresponding to the object of operation can be directly confirmed.

[0118] When the voice text is a long sentence that cannot be directly matched to the current application interface, execute S940.

[0119] S940 performs semantic word segmentation on the speech text obtained by S920 based on the recognition results of S900 (e.g., interface OCR text recognition results, icon semantic recognition results) to obtain the word segmentation results.

[0120] S941, based on the word segmentation results, determine the operation object and operation method pointed to by the voice text. The operation object is the application interface content other than the operable controls on the current application interface.

[0121] Specifically, long sentences typically employ a "verb + object" structure, such as "open the folder" or "give xx a thumbs up." Therefore, in S940, during word segmentation, the text content, icon meanings, image attributes, and lexical attributes from the current application interface recognition results are incorporated into the word segmentation algorithm (e.g., Natural Language Processing (NLP) word segmentation algorithm) to extract and correct verbs and objects from the speech text, thereby obtaining the execution relationship of the operation.

[0122] For example, targeting Figure 5 The interface shown is as follows. When the user inputs the voice message "Give user C a thumbs up", the voice text "Give user C a thumbs up" is segmented into three parts: "give", "user C", and "thumbs up" based on the text recognition result "user C" and the icon recognition result "thumbs up" on the application interface.

[0123] S942, based on the grouping results of S910, search and match the operation objects in the word segmentation results obtained in S940 to confirm the operable controls corresponding to the operation objects pointed to by the voice text.

[0124] When the operation object in S941 is an operable control on the current application interface, S931 can be skipped and the operable control corresponding to the operation object can be directly confirmed.

[0125] S950 generates operation commands corresponding to user voice input based on the operation method and operable controls corresponding to the voice text.

[0126] S960 inputs operation commands into the first application, causing the first application to execute the operation commands, thereby realizing voice operation for the first application.

[0127] Preferably, in one embodiment, in S900, a combination of implementation methods 2.1, 2.2, and 2.3 is used. Specifically, the OCR text recognition results (text content and text box positions) and icon recognition results (icon meaning and icon position) of the GUI interface are obtained, and the types and positions of the text controls, icon controls, and image controls of the interface are obtained by parsing from the runtime XML file of the smart terminal system.

[0128] In S910, a combination of implementation methods 3.1, 3.2 and 3.3 is adopted.

[0129] Specifically, for grouping logic 1, based on the positional constraints of icon + text, image + text, and text, icon, and image of container controls, a heuristic relationship recognition rule is designed to detect the relationship between controls and obtain the first relationship group.

[0130] For grouping logic 2, on the one hand, the runtime XML file is parsed, and backtracking is performed from the leaf node to the next higher level node. Based on the backtracking termination condition, the association relationship between the page list and logical grouping is obtained. On the other hand, by combining the control types and positions based on image detection and XML parsing, the model is trained based on the control association annotation data to obtain the control cross-association strength matrix, and to obtain the second association relationship grouping for the association relationship of the interface list class and the logical subordinate class grouping.

[0131] Figure 10 The diagram shown is a flowchart illustrating the execution of a method according to an embodiment of this application. During the operation of the second application, the electronic device 100 performs the following... Figure 10 The following method flow is shown.

[0132] S1010: Obtain the OCR text recognition results (recognized text and text box positions) and runtime xml file of the current application interface.

[0133] S1020: Parse the runtime XML file to obtain the type and position of the UI controls in the current application interface (save them in the form of a structure).

[0134] S1030, design heuristic grouping rules based on the positional relationship of interface controls, group interface controls based on heuristic grouping rules (grouping logic 2.1, implementation method 3.1), and obtain the grouping result (first association relationship grouping).

[0135] S1040: Group the UI controls based on the node tree in the runtime XML file and obtain the grouping results (second association relationship grouping).

[0136] S1050, performs word segmentation on the semantic ASR results of the user's input speech. Refer to S930.

[0137] S1060, based on the first and second association relationship groupings, perform a search and match on the word segmentation results to confirm the operable control corresponding to the operation object pointed to by the speech text. Refer to S940.

[0138] S1070: Based on the operation method of the voice-text pointing and the operable control corresponding to the operation object pointed to by the voice-text, generate the corresponding operation command for the user's voice input. See S950.

[0139] S1080: The operation command is input to the first application, causing the first application to execute the operation command, thereby realizing voice operation for the first application. See S960.

[0140] According to the method of the embodiments of this application, this application also proposes a voice operation device installed in an electronic device (e.g., electronic device 100).

[0141] Figure 11 The diagram shown is a simplified structural diagram of a voice operation device according to an embodiment of this application. Figure 11 As shown, the device 1100 includes: The recognition module 1110 is used to perform content recognition on the current application interface and identify content elements on the current application interface; (refer to S900) Grouping module 1120 is used to group content elements on the current application interface; (refer to S900) The text acquisition module 1130 is used to acquire the voice text corresponding to the user's input voice. The word segmentation module 1131 is used to perform semantic word segmentation on the speech text based on the recognition result of the recognition module 1110 when the speech text is a long sentence that cannot be directly matched to the current application interface, obtain the word segmentation result, and confirm the operation object and operation method pointed to by the speech text; (refer to S940). The lookup and matching module 1140 is used to, when the voice text is an executable phrase that can be directly matched to the current application interface, perform a lookup and matching on the operation object pointed to by the voice text according to the grouping results, and confirm the operable control corresponding to the operation object pointed to by the voice text; (see S930); and, according to the grouping results, perform a lookup and matching on the operation objects in the word segmentation results of the word segmentation module 1131, and confirm the operable control corresponding to the operation object pointed to by the voice text; (see S941). The operation instruction generation module 1150 is used to generate operation instructions corresponding to the user's voice input based on the operation method and operable controls corresponding to the voice text. (Refer to S950) In the description of the embodiments of this application, for the sake of convenience, the device is described by dividing it into various modules according to its functions. The division of each module is only a logical functional division. When implementing the embodiments of this application, the functions of each module can be implemented in one or more software and / or hardware.

[0142] Specifically, the apparatus proposed in this application can be fully or partially integrated onto a single physical entity, or physically separated. These modules can be implemented entirely in software via processing element calls; entirely in hardware; or partially in software via processing element calls and partially in hardware. For example, the detection module can be a separate processing element or integrated into a chip in the electronic device. The implementation of other modules is similar. Furthermore, these modules can be fully or partially integrated together, or implemented independently. During implementation, each step of the above method or each of the above modules can be completed through integrated logic circuits in the hardware of the processor element or through software instructions.

[0143] For example, these modules can be one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (ASICs), one or more digital signal processors (DSPs), or one or more field-programmable gate arrays (FPGAs). Alternatively, these modules can be integrated together as a system-on-a-chip (SOC).

[0144] In practical applications, the method flow of the embodiments shown in this specification can be implemented by an electronic chip installed on an electronic device. Therefore, one embodiment of this application proposes an electronic chip. For example, the electronic chip is installed on an electronic device and includes: A processor is used to execute computer program instructions stored in memory, wherein when the computer program instructions are executed by the processor, an electronic chip is triggered to perform the steps of the method described in the embodiments of this application.

[0145] An embodiment of this application also proposes an electronic device (electronic device 100), which includes a memory for storing computer program instructions and a processor for executing the program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the steps of the method described in the embodiment of this application.

[0146] Specifically, in one embodiment of this application, the one or more computer programs are stored in the memory, and the one or more computer programs include instructions that, when executed by the device, cause the device to perform the method steps described in the embodiment of this application.

[0147] For example, Figure 12 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application is shown. Figure 12 As shown, the electronic device 1200 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) interface 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, a headphone jack 170D, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195, etc. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, a barometric pressure sensor 180C, a magnetic sensor 180D, an accelerometer sensor 180E, a distance sensor 180F, a proximity sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0148] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 1200. In other embodiments of this application, the electronic device 1200 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0149] The processor 110 of the electronic device 1200 can be an on-chip device (SOC), which may include a central processing unit (CPU) or other types of processors. For example, the processor 110 may be a PWM control chip.

[0150] The processor 110 may include, for example, a CPU, DSP, microcontroller, or digital signal processor, and may also include a GPU, embedded neural network processing units (NPU), and image signal processor (ISP). The processor 110 may also include necessary hardware accelerators or logic processing hardware circuitry, such as an ASIC, or one or more integrated circuits for controlling the execution of the program in this application. Furthermore, the processor 110 may have the function of operating one or more software programs, which may be stored in a storage medium.

[0151] Processor 110 may include one or more processing units. For example, processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent components or integrated into one or more processors. In some embodiments, electronic device 1200 may also include one or more processors 110. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0152] In some embodiments, the processor 110 may include one or more interfaces. These interfaces may include an inter-integrated circuit (I2C) interface, an integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a SIM card interface, and / or a USB interface, etc. The USB interface 130 is a USB standard-compliant interface, specifically a Mini USB interface, a Micro USB interface, a USB Type-C interface, etc. The USB interface 130 can be used to connect a charger to charge the electronic device 1200, and can also be used for data transfer between the electronic device 1200 and peripheral devices.

[0153] It is understood that the interface connection relationships between the modules illustrated in the embodiments of this application are merely illustrative and do not constitute a structural limitation on the electronic device 1200. In other embodiments of this application, the electronic device 1200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0154] The external storage interface 120 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 1200. The external memory card communicates with the processor 110 through the external storage interface 120 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0155] The internal memory 121 of the electronic device 1200 can be used to store one or more computer programs, which include instructions. The processor 110 can execute the instructions stored in the internal memory 121, thereby causing the electronic device 1200 to perform the methods provided in some embodiments of this application, as well as various applications and data processing. The internal memory 121 may include a code storage area and a data storage area. The code storage area may store the operating system. The data storage area may store data created during the use of the electronic device 1200. Furthermore, the internal memory 121 may include high-speed random access memory and may also include non-volatile memory, such as one or more disk storage components, flash memory components, universal flash storage (UFS), etc.

[0156] The internal memory 121 may be a read-only memory (ROM), other types of static storage devices that can store static information and instructions, random access memory (RAM), or other types of dynamic storage devices that can store information and instructions. It may also be an electrically erasable programmable read-only memory (EEPROM), a compact disc read-only memory (CD-ROM) or other optical disc storage, optical disc storage (including compressed optical discs, laser discs, optical discs, digital universal optical discs, Blu-ray discs, etc.), a magnetic disk storage medium, or other magnetic storage devices. Alternatively, it may be any computer-readable medium capable of carrying or storing desired program code in the form of instructions or data structures that can be accessed by a computer.

[0157] Processor 110 and internal memory 121 can be combined into a single processing device, or more commonly, they are independent components. Processor 110 is used to execute program code stored in internal memory 121 to implement the methods described in the embodiments of this application. In specific implementations, internal memory 121 can also be integrated into the processor, or it can be independent of the processor.

[0158] The charging management module 140 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 140 receives charging input from the wired charger via a USB interface 130. In some wireless charging embodiments, the charging management module 140 receives wireless charging input via the wireless charging coil of the electronic device 1200. While charging the battery 142, the charging management module 140 can also supply power to the electronic device 1200 via the power management module 141.

[0159] The power management module 141 connects the battery 142, the charging management module 140, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 140, and supplies power to the processor 110, internal memory 121, external memory interface 120, mobile communication module 150, and wireless communication module 160. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 140 may be located in the same device.

[0160] The wireless communication function of electronic device 1200 can be realized through antenna 1, antenna 2, mobile communication module 150, wireless communication module 160, modem processor and baseband processor, etc.

[0161] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 1200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0162] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 1200. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110.

[0163] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 194. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0164] The wireless communication module 160 can provide solutions for wireless communication applications on the electronic device 1200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 160 can be one or more devices integrating at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 160 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0165] In some embodiments, antenna 1 of electronic device 1200 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 160, enabling electronic device 1200 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0166] Electronic device 1200 can perform shooting functions through ISP, camera 193, video codec, GPU, display 194 and application processor.

[0167] Electronic device 1200 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0168] Electronic device 1200 can implement touch input through display screen 194, touch sensor 180K, and processor 110. For example, touch sensor 180K and display screen 194 are integrated into a touch screen. The user's click operation on the touch screen is collected by touch sensor 180K as a touch signal. The touch signal is collected and converted by sensor module 180 and then transmitted to processor 110. Processor 110 identifies and analyzes the touch signal to determine the user's touch operation behavior.

[0169] In one embodiment, Figure 4 The touch panel shown can be made by Figure 12 The touch sensor 180K shown can be used to implement the touch detection module, or it can be implemented by combining the touch sensor 180K with the processing chip in the sensor module 180. The touch detection module can be implemented based on the sensor module 180, or it can be implemented by the processor 110, or it can be implemented by hardware modules other than the sensor module 180 and the processor 110.

[0170] Furthermore, the devices, apparatuses, and modules described in the embodiments of this application can be implemented by computer chips or physical entities, or by products with certain functions.

[0171] Those skilled in the art will understand that embodiments of this application can be provided as methods, apparatus, or computer program products. Therefore, the present invention can take the form of a completely hardware embodiment, a completely software embodiment, or an embodiment combining software and hardware aspects. Furthermore, the present invention can take the form of a computer program product embodied on one or more computer-usable storage media containing computer-usable program code.

[0172] In the several embodiments provided in this application, any function, if implemented as a software functional unit and sold or used as an independent product, can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, a server, or a network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application.

[0173] Specifically, one embodiment of this application also provides a computer-readable storage medium storing a computer program that, when run on a computer, causes the computer to execute the method provided in the embodiment of this application.

[0174] An embodiment of this application also provides a computer program product, which includes a computer program that, when run on a computer, causes the computer to perform the method provided in the embodiment of this application.

[0175] The embodiments described in this application are described with reference to flowchart illustrations and / or block diagrams of methods, apparatus (devices), and computer program products according to embodiments of this application. It will be understood that each block of the flowchart illustrations and / or block diagrams, and combinations of blocks in the flowchart illustrations and / or block diagrams, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, special-purpose computer, embedded processor, or other programmable data processing apparatus to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing apparatus, generate instructions for implementing the flowchart illustrations. Figure 1 One or more processes and / or boxes Figure 1 A device that provides the functions specified in one or more boxes.

[0176] These computer program instructions may also be stored in a computer-readable storage medium that can direct a computer or other programmable data processing device to function in a particular manner, such that the instructions stored in the computer-readable storage medium produce an article of manufacture including instruction means, which are implemented in a process Figure 1 One or more processes and / or boxes Figure 1 The function specified in one or more boxes.

[0177] These computer program instructions may also be loaded onto a computer or other programmable data processing equipment to cause a series of operational steps to be performed on the computer or other programmable equipment to produce a computer-implemented process, thereby providing instructions that execute on the computer or other programmable equipment for implementing the process. Figure 1 One or more processes and / or boxes Figure 1 The steps of the function specified in one or more boxes.

[0178] It should also be noted that in the embodiments of this application, "at least one" refers to one or more, and "more than one" refers to two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent the existence of A alone, the simultaneous existence of A and B, or the existence of B alone. A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one of the following" and similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, and c can represent: a, b, c, a and b, a and c, b and c, or a and b and c, where a, b, and c can be single or multiple.

[0179] In this application embodiment, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. Without further limitations, an element defined by the phrase "comprising one..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0180] This application can be described in the general context of computer-executable instructions, such as program modules, that are executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform a specific task or implement a specific abstract data type. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0181] The various embodiments in this application are described in a progressive manner. Similar or identical parts between embodiments can be referred to mutually. Each embodiment focuses on describing the differences from other embodiments. In particular, the device embodiments are basically similar to the method embodiments, so the description is relatively simple; relevant parts can be referred to the descriptions of the method embodiments.

[0182] Those skilled in the art will recognize that the units and algorithm steps described in the embodiments of this application can be implemented using electronic hardware, computer software, or a combination of electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0183] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the specific working processes of the devices, apparatuses, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0184] The above description is merely a specific embodiment of this application. Any variations or substitutions that can be easily conceived by those skilled in the art within the scope of the technology disclosed in this application should be included within the protection scope of this application. The protection scope of this application should be determined by the protection scope of the claims.< / book> < / book> < / book> < / book> < / book> < / book>

Claims

1. A voice control method, the method being applied to an electronic device, characterized in that, The method includes: Identify content elements on the current application interface, including operable controls and application interface content other than the operable controls; The content elements on the current application interface are grouped, and the operable controls with related relationships are grouped together with the application interface content other than the operable controls. The grouping results are obtained. Among them, the content elements belonging to the same list item in the list arrangement on the current application interface are grouped together. Get the speech text corresponding to the user's input speech; Based on the grouping results, the operable controls corresponding to the operation objects are identified, wherein the operation objects are the content elements pointed to by the voice text; For the operable controls corresponding to the operation object, generate operation instructions corresponding to the user's voice input.

2. The method according to claim 1, characterized in that, The identification of content elements on the current application interface includes: Obtain the Extensible Markup Language (XML) file of the electronic device during runtime, and parse the XML file to obtain the types and positions of the text controls, icon controls, and image controls of the current application interface.

3. The method according to claim 1, characterized in that, The identification of content elements on the current application interface includes: Perform Optical Character Recognition (OCR) on the current application interface to obtain the OCR recognition result.

4. The method according to claim 1, characterized in that, The identification of content elements on the current application interface includes: Perform icon recognition on the icons on the current application interface and obtain the icon recognition results.

5. The method according to claim 1, characterized in that, The identification of content elements on the current application interface includes: Obtain a screenshot of the current application interface, and perform image recognition on the screenshot using an application interface recognition model to identify operable controls on the current application interface.

6. The method according to claim 1, characterized in that, The step of grouping the content elements on the current application interface and obtaining the grouping results includes: Group the operable controls in the content element with their corresponding text descriptions.

7. The method according to claim 6, characterized in that, The step of grouping the content elements on the current application interface and obtaining the grouping results includes: The association relationship of the content elements on the current application interface is detected according to the preset association relationship recognition rules. The association relationship between the content elements is confirmed, and the content elements with the association relationship are grouped together.

8. The method according to claim 7, characterized in that, The method further includes: Based on the display scheme configuration of the application interface and the control logic of the operable controls, a heuristic relationship recognition rule is designed.

9. The method according to claim 6, characterized in that, The step of grouping the content elements on the current application interface and obtaining the grouping results includes: Based on the control tree in the XML file of the current application interface, determine the relationship between the controls corresponding to each node in the control tree; The controls in the content element are grouped according to the relationships between them.

10. The method according to claim 6, characterized in that, The step of grouping the content elements on the current application interface and obtaining the grouping results includes: The content elements of the current application interface are grouped using a grouping model.

11. The method according to any one of claims 1-10, characterized in that, The step of confirming the operable controls corresponding to the operation object based on the grouping results includes: For the voice text that is a short sentence, identify the operation object that the voice text points to, wherein the operation object is application interface content other than the operable control; Based on the grouping results, the operation object is searched and matched to confirm the operable control corresponding to the operation object.

12. The method according to any one of claims 1-10, characterized in that, The step of confirming the operable controls corresponding to the operation object based on the grouping results includes: For the speech text that is a long sentence, semantic word segmentation is performed on the speech text based on the grouping results to obtain the word segmentation results; The operation object pointed to by the voice text is determined based on the word segmentation result, wherein the operation object is the application interface content other than the operable control; Based on the grouping results, the operation object is searched and matched to confirm the operable control corresponding to the operation object.

13. A voice control device, said device being applied to an electronic device, characterized in that, The device includes: The identification module is used to identify content elements on the current application interface, including operable controls and application interface content other than the operable controls. The grouping module is used to group the content elements on the current application interface, grouping the operable controls with related relationships with the application interface content other than the operable controls into a group, and obtaining the grouping result; the grouping of the content elements on the current application interface and obtaining the grouping result includes: grouping the content elements belonging to the same list item in the list arrangement on the current application interface into a group. The text acquisition module is used to acquire the voice text corresponding to the user's input voice. The search and matching module is used to identify the operable control corresponding to the operation object based on the grouping results, wherein the operation object is the content element pointed to by the voice text; The instruction generation module is used to generate operation instructions corresponding to the user's voice input for the operable controls corresponding to the operation object.

14. An electronic device, characterized in that, The electronic device includes a memory for storing computer program instructions and a processor for executing the computer program instructions, wherein when the computer program instructions are executed by the processor, the electronic device is triggered to perform the method steps as described in any one of claims 1-12.

15. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program that, when run on a computer, causes the computer to perform the method as described in any one of claims 1-12.

Images