Voice interaction method, electronic device, and computer-readable storage medium

Abstract

This application discloses a voice interaction method, an electronic device, and a computer-readable storage medium for resolving control matching conflicts. The method includes: the electronic device processing acquired first voice to obtain a first voice command; acquiring text description information and position information of each first control within the current interface; determining the visual focus area of ​​the human eye on the screen; matching the first voice command with target information, the target information including the text description information of each first control; if the current interface includes at least two first target controls, for each first target control, determining whether the first target control is located within the visual focus area based on the position information of the first target control, wherein the first target control is the control whose text description information matches the first voice command; when the visual focus area includes only one first target control, performing a preset operation on the first target control within the visual focus area.

Description

Technical Field

[0001] This application relates to the field of human-computer interaction technology, and in particular to a voice interaction method, electronic device, computer-readable storage medium, and computer program product. Background Technology

[0002] With the continuous development of human-computer interaction technology, the application of voice control of electronic devices is becoming more and more widespread.

[0003] Currently, the process of controlling electronic devices via voice can be as follows: The user inputs their voice into the electronic device; the electronic device collects the user's voice, recognizes it, and obtains the voice command; the electronic device traverses the interface layout file of the current display interface to obtain the text description and coordinate information of each system native control on the current display interface; then, it matches the voice command with the text description information of each control to find the control that matches the voice command; finally, it performs a click operation on the control that matches the voice command to realize voice control of the electronic device.

[0004] If the current display interface includes at least two controls with identical text descriptions, and the voice command matches both of these controls (i.e., at least two controls match the voice command), a control matching conflict occurs. Control matching conflicts may result in inaccurate matching of user intent, leading to mismatches. Summary of the Invention

[0005] This application provides a voice interaction method, an electronic device, a computer-readable storage medium, and a computer program product that can resolve control matching conflicts.

[0006] In a first aspect, embodiments of this application provide a voice interaction method applied to an electronic device. The method includes: acquiring a first voice; processing the first voice to obtain a first voice command; acquiring text description information and position information of each first control within a current interface, wherein the first control is a control belonging to a first category, and the current interface displayed on the screen of the electronic device includes at least one control; determining the visual focus area of ​​the human eye on the screen; matching the first voice command with target information, wherein the target information includes text description information of each first control; if the current interface includes at least two first target controls, for each first target control, determining whether the first target control is located within the visual focus area based on the position information of the first target control, wherein the first target control is a control whose text description information matches the first voice command; and when the visual focus area includes only one first target control, performing a preset operation on the first target control within the visual focus area.

[0007] As can be seen above, when there are at least two first target controls that match the first voice command, i.e., when a control matching conflict occurs, the visual focus area is used to filter the at least two first target controls. If the visual focus area includes only one first target control, then the first target control within the visual focus area is determined to be the control that matches the first voice command. In this way, by reducing the control matching range through the visual focus area, the possibility of matching conflicts is reduced, and the matching accuracy is improved when control matching conflicts occur.

[0008] In some possible implementations of the first aspect, when the visual focus area includes at least two first target controls, the method further includes: displaying a unique identifier for each first target control within the visual focus area; acquiring a second voice; processing the second voice to obtain a second voice instruction; matching the second voice instruction with the unique identifier of each first target control; and when there is a second target control within the visual focus area, performing a preset operation on the second target control, wherein the second target control is a first target control whose unique identifier matches the second voice instruction.

[0009] In this implementation, when the visual focus area includes at least two primary target controls, a unique identifier is added to each primary target control to allow the user to reconfirm their control intent, further improving the matching accuracy when control matching conflicts occur. Additionally, displaying only the unique identifiers of the primary target controls within the visual focus area provides a better user experience.

[0010] In some possible implementations of the first aspect, after displaying a unique identifier for each first target control within the visual focus area and before acquiring the second voice, the method further includes: displaying a prompt message to prompt for voice input in response to the unique identifier.

[0011] In this implementation, after displaying the unique identifier of the first target control, the electronic device prompts the user to input the control voice again through a prompt message, resulting in a better user experience.

[0012] In some possible implementations of the first aspect, the target information mentioned above also includes text description information of each second control within the visual focus area, wherein the second control is a control belonging to the second category;

[0013] Before matching the first voice command with the target information, the method further includes: traversing the current interface's page layout file to obtain the position information and control type information of each control; determining whether a second control is included within the visual focus area based on the position information and control type information of each control; and when at least one second control is included within the visual focus area, performing optical character recognition (OCR) on the visual focus area to obtain the OCR result, which includes the text description information of each second control within the visual focus area. A second control refers to a control whose text description information cannot be obtained by traversing the interface layout file, such as a WebView control.

[0014] In this implementation, when the visual focus area includes a second control, optical character recognition (OCR) is performed on the visual focus area to obtain the text description information of the second control, which improves the control recognition coverage and further improves the accuracy of control matching.

[0015] Among some possible implementations of the first aspect, the second category includes WebView controls and / or third-party custom controls, while the first category includes system native controls.

[0016] In some possible implementations of the first aspect, the process of determining the visual focus area of ​​the human eye's gaze on the screen may include: obtaining at least two candidate visual focus areas by performing at least two gaze focus area estimations, wherein the candidate visual focus areas are the visual focus areas of the human eye's gaze on the screen; matching the first voice command with the text description information of each first control; when there is at least one third target control, and none of the candidate visual focus areas include the third target control, taking the area formed by the leftmost boundary line, rightmost boundary line, topmost boundary line, and bottommost boundary line of the at least two candidate visual focus areas as the visual focus area, and the third... The target control is a first control whose text description information matches the first voice command; when there is at least one third target control, and the at least one third target control is not located in the intersection area of ​​at least two candidate visual focus areas, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line, and the bottommost boundary line of the at least two target candidate visual focus areas is taken as the visual focus area, and the target candidate visual focus area is the candidate visual focus area including the third target control; when there is at least one third target control, and the at least one third target control is located in the intersection area of ​​at least two target candidate visual focus areas, the intersection area is taken as the visual focus area.

[0017] In this implementation, by combining multiple estimated visual focus areas, voice commands, and text descriptions of controls, the final user visual focus area is determined based on the situation of the third target control matching the first voice command in each estimated visual focus area, thereby improving the accuracy of eye tracking.

[0018] In some possible implementations of the first aspect, if the current interface includes only one first target control, the method further includes: performing a preset operation on the first target control.

[0019] Secondly, embodiments of this application provide a voice interaction method applied to an electronic device. The method includes: acquiring a first voice; processing the first voice to obtain a first voice command; determining the visual focus area of ​​a human eye's gaze on the screen of the electronic device, wherein the current interface displayed on the screen includes at least one control; acquiring text description information of each first control, wherein the first control is a control belonging to a first category and located within the visual focus area; matching the first voice command with target information, wherein the target information includes the text description information of each first control; and performing a preset operation on the first target control when a first target control exists within the visual focus area, wherein the first target control is a control whose text description information matches the first voice command.

[0020] In some possible implementations of the second aspect, when there are at least two first target controls within the visual focus area, the method further includes: displaying a unique identifier for each first target control; acquiring a second voice; processing the second voice to obtain a second voice instruction; matching the second voice instruction with the unique identifier of each first target control; and when there is a second target control within the visual focus area, performing a preset operation on the second target control, wherein the second target control is a first target control whose unique identifier matches the second voice instruction.

[0021] In some possible implementations of the second aspect, the target information also includes text descriptions of each second control within the visual focus area, wherein the second control is a control belonging to a second category; before matching the first voice command with the target information, the method further includes: traversing the page layout file of the current interface to obtain the position information and control type information of each control; determining whether the visual focus area includes a second control based on the position information and control type information of each control; if the visual focus area includes at least one second control, then performing optical character recognition on the visual focus area to obtain an optical character recognition result, wherein the optical character recognition result includes text description information of each second control within the visual focus area.

[0022] In some possible implementations of the second aspect, the process of determining the visual focus area of ​​the human eye's gaze on the screen of the electronic device may include: obtaining at least two candidate visual focus areas by performing at least two gaze focus area estimations, wherein the candidate visual focus areas are the visual focus areas of the human eye's gaze on the screen; matching the first voice command with the text description information of each first control; and when there is at least one third target control, and none of the candidate visual focus areas include the third target control, taking the area formed by the leftmost boundary line, rightmost boundary line, topmost boundary line, and bottommost boundary line of the at least two candidate visual focus areas as the visual focus area. The third target control is the first control whose text description information matches the first voice command; when there is at least one third target control, and at least one third target control is not located in the intersection area of ​​at least two candidate visual focus areas, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of at least two candidate visual focus areas is taken as the visual focus area, and the candidate visual focus area is the candidate visual focus area including the third target control; when there is at least one third target control, and at least one third target control is located in the intersection area of ​​at least two candidate visual focus areas, the intersection area is taken as the visual focus area.

[0023] Thirdly, embodiments of this application provide an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the computer program to implement the method as described in either the first or second aspect above.

[0024] Fourthly, embodiments of this application provide a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method as described in either the first or second aspect above.

[0025] Fifthly, embodiments of this application provide a chip system including a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the method described in either the first or second aspect above. The chip system may be a single chip or a chip module composed of multiple chips.

[0026] Sixthly, embodiments of this application provide a computer program product that, when run on an electronic device, causes the electronic device to perform the method described in either the first or second aspect above.

[0027] It is understood that the beneficial effects of the second to sixth aspects mentioned above can be found in the relevant descriptions in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0028] Figure 1 A schematic diagram of a scenario for a voice-controlled electronic device provided in an embodiment of this application;

[0029] Figure 2 Another schematic diagram of the voice-controlled electronic device provided in the embodiments of this application;

[0030] Figure 3 A schematic diagram of the current display interface of the large-screen device 22 provided in this application embodiment;

[0031] Figure 4 Another schematic diagram of the current display interface of the large-screen device 22 provided in the embodiments of this application;

[0032] Figure 5 A schematic diagram of the prompting information provided in the embodiments of this application;

[0033] Figure 6 A flowchart illustrating a voice interaction method provided in an embodiment of this application;

[0034] Figure 7 Another flowchart illustrating the voice interaction method provided in this application embodiment;

[0035] Figure 8 A schematic diagram illustrating the determination of the visual focus area provided in an embodiment of this application;

[0036] Figure 9 Another schematic diagram for determining the visual focus area provided in this application embodiment;

[0037] Figure 10 Another schematic diagram for determining the visual focus area provided in this application embodiment;

[0038] Figure 11 Another schematic flowchart of the voice interaction method provided in the embodiments of this application;

[0039] Figure 12 A schematic block diagram of a voice interaction device provided in an embodiment of this application;

[0040] Figure 13 A flowchart illustrating the process of a voice interaction device provided in this application embodiment;

[0041] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0042] In the following description, specific details such as particular system architectures and technologies are set forth for illustrative purposes and not for limiting purposes, in order to provide a thorough understanding of the embodiments of this application.

[0043] The following provides an exemplary description of the application scenarios that may be involved in the embodiments of this application.

[0044] See Figure 1 This is a schematic diagram of a scenario for a voice-controlled electronic device provided in an embodiment of this application. Figure 1 As shown, this scenario includes user 11 and large screen device 12. The current display interface 121 of large screen device 12 includes controls 122, 123, 124, 125, 126, 127, 128 and 129.

[0045] User 11 inputs the voice command "Open movie" into the large-screen device 12. After the large-screen device 12 collects the user's voice through a microphone, it processes the collected voice to obtain the corresponding voice command. For example, the large-screen device 12 can collect the user's voice through a microphone array and process it through Automatic Speech Recognition (ASR) to obtain a voice control command.

[0046] The large-screen device 12 can obtain the text description information and position information of each system native control on the current display interface 121 by traversing the interface layout file.

[0047] The text description information of a control is used to characterize the textual semantics of the control. For example, if the text displayed by control 122 is "Homepage", then the text description information of control 122 includes the word "Homepage". Similarly, the text description information of control 123 includes the word "Movies".

[0048] The position information of a control is used to characterize its position on the interface or screen, and is usually expressed in the form of coordinates. For example, the position information of control 122 can be used to determine the specific location of control 122 on the current display interface 121.

[0049] After obtaining the voice command "Open Movie" and the text description information of each system native control on the current interface 121, the large screen device 12 matches the voice command with the text description information of each system native control to find the control that matches the voice command.

[0050] For example, the word "movie" in the voice command is matched with the text description information of each control on the current interface 121. If the text description information of control 123 includes "movie" and matches "movie" in the voice command, then control 123 is determined to be the control that matches the voice command.

[0051] The large-screen device 12 matches voice commands with control text descriptions to identify the control that matches the voice command, and then performs the corresponding operation on that control. For example, if the control that matches the voice command "Open Movie" is control 123, then a simulated click operation is performed on control 123.

[0052] In this way, user 11 was able to control the large-screen device 12 via voice.

[0053] It is understandable that the text descriptions of controls may be the same or different. Therefore, if two controls with the same text description exist on the current display interface 121, two controls will be matched to the voice command, resulting in a control matching conflict. When a control matching conflict occurs, the large-screen device 12 cannot determine which control the user actually wants to match, which may lead to mismatches.

[0054] For example, both controls 123 and 128 are native system controls. The text descriptions of controls 123 and 128 can be obtained by traversing the interface layout file of the currently displayed interface 121. Assume that the text descriptions of both controls 123 and 128 include the word "movie". In this case, the controls matching the user's voice command "open movie" include controls 123 and 128. The large-screen device 12 cannot determine whether the user's voice command "open movie" is directed at control 123 or control 128.

[0055] In addition to the text descriptions of the system's native controls, the interface layout file also includes control type information for each control on the currently displayed interface 121. Control type information is used to characterize the type of the control. Examples of control types include system native controls, third-party custom controls (non-system native controls), and WebView controls.

[0056] While iterating through the UI layout file can retrieve text descriptions of native system controls, it cannot retrieve text descriptions of third-party custom controls and WebView controls. Without access to text descriptions of third-party custom controls and WebView controls, these controls cannot be matched with user voice commands.

[0057] For example, assuming that the control 129 on the current display interface 121 is a WebView control, the large screen device 12 cannot obtain the text description information of the control 129 by traversing the interface layout file, and therefore cannot match the voice command with the text description information of the control 129.

[0058] To address the issues of control matching conflicts and low control recognition coverage in scenarios where electronic devices are controlled by voice, this application provides a multi-modal interaction solution that combines vision and voice.

[0059] In this embodiment of the application, if there are at least two controls that match the voice command on the current display interface, the user's eye is used to filter the at least two matching controls in the visual focus area on the screen to reduce the range of control matching, reduce the possibility of control matching conflicts, and improve the accuracy of control matching.

[0060] For example, the following is combined with Figures 2 to 5 This section introduces and explains the voice interaction process that combines vision and speech.

[0061] in, Figure 2 This is another schematic diagram of a voice-controlled electronic device provided in an embodiment of this application. Figure 3 This is a schematic diagram of the current display interface of the large-screen device 22 provided in an embodiment of this application. Figure 4 Another schematic diagram of the current display interface of the large-screen device 22 provided in the embodiments of this application. Figure 5 This is a schematic diagram of the prompt information provided in an embodiment of this application.

[0062] like Figure 2 As shown, user 21 is viewing the interface 221 currently displayed on the large screen device 22, and the area of ​​the user's gaze on the screen of the large screen device 22 is the visual focus area 222. Furthermore, user 21 inputs the voice command "Open game A" into the large screen device 22.

[0063] The interface 221 currently displayed on the large screen device 22 can be as follows: Figure 3 As shown. In Figure 3 In this interface 221, controls 223 to 234 are included, each with corresponding text. For example, the text on control 223 is "Game B", the text on control 224 is "Game A", and the text on control 225 is "Game B". Interface 221 can be the interface of a video playback software, and each control on this interface can correspond to a video or a live stream. For example, control 224 can correspond to the video of Game A or the live stream of Game A.

[0064] The large-screen device 22 can integrate a microphone array and a camera. The large-screen device 22 uses the microphone array to capture the user's voice command "Open game A," and simultaneously uses the camera to capture the user's facial image. After capturing the user's voice, it performs ASR processing to obtain the voice command. After capturing the facial image, the large-screen device 22 performs facial region detection to determine the facial region in the image; then, it locates the pupil center based on the facial region; next, based on the correspondence between the image coordinate system and the screen coordinate system of the large-screen device 22, and the pupil center in the image coordinate system, it calculates the coordinates of the human eye on the screen, thereby determining the visual focus area 222 of the human eye's gaze on the screen.

[0065] The large-screen device 22 obtains the coordinates and text descriptions of the native system controls on the interface 221 by traversing the interface layout file of the interface 221.

[0066] In some embodiments, after obtaining the text description information of the system's native controls, the large-screen device 22 matches the voice commands with the text description information of each system's native controls.

[0067] In other embodiments, to improve control recognition accuracy, the large-screen device 22, in addition to obtaining the text description information of each system native control by traversing the interface layout file, can also obtain the control type information of each control from the interface layout file. Based on the control type information, it determines whether the visual focus area 222 includes third-party custom controls and WebView controls, etc. If the visual focus area 222 includes at least one of third-party custom controls and WebView controls, then OCR recognition is performed on the visual focus area 222 to obtain the OCR recognition result. After obtaining the text description information of the system native controls and the OCR recognition result of the visual focus area, the large-screen device 22 matches the voice command with the text description information and OCR recognition result of the system native controls.

[0068] If only one control in interface 221 matches the voice command "Open Game A", then click on that control.

[0069] If at least two controls in interface 221 match the voice command "Open Game A", then based on the coordinates of these two controls, it is determined whether the controls are located in the visual focus area 222, and the controls located in the visual focus area 222 are counted. If the visual focus area includes only one control that matches the voice command, then a click operation is performed on the control in the visual focus area; if the visual focus area includes at least two controls that match the voice command, then the unique identifier of each control in the visual focus area that matches the voice command is displayed, and multiple rounds of dialogue are conducted with the user to reconfirm the user's control intention.

[0070] exist Figure 3 In the interface 221 shown, the controls that match the voice command "Open Game A" include controls 224, 226, 228, and 233. That is, there are at least two controls that match the voice command, resulting in a control matching conflict.

[0071] Furthermore, based on the coordinate information of controls 226, 228, and 233, the large-screen device 22 determines whether these three controls are located within the visual focus area 222, and counts the number of controls located within the visual focus area 222. At this point, if the large-screen device 22 determines that control 224 is located within the visual focus area 222, and that the visual focus area 222 contains only one control that matches the voice command, then a click operation is performed on control 224.

[0072] And in Figure 4 In the interface 221 shown, the controls that match the voice command "Open Game A" include controls 224, 225, 226, 228 and 233. That is, there are at least two controls that match the voice command, resulting in a control matching conflict.

[0073] The large-screen device 22 determines whether each matching control is located within the visual focus area 222 based on the coordinates of the matching controls and the coordinates of the visual focus area 222, and counts the number of matching controls located within the visual focus area 222. At this time, controls 224 and 225 are located within the visual focus area 222, meaning that the visual focus area 222 includes at least two controls that match the voice command.

[0074] To further confirm the user's control intent, the large-screen device 22 can display a unique identifier for each control within the visual focus area 222 that matches the voice command, and the user can input control voice again based on this unique identifier.

[0075] exist Figure 4 In the process, after the large-screen device 22 determines that the visual focus area 222 includes controls 224 and 225, it displays a subscript 235 for control 224 and a subscript 236 for control 225. Subscript 235 serves as the unique identifier for control 224, and subscript 236 serves as the unique identifier for control 225.

[0076] After the large-screen device 22 displays badges 235 and 236, the user 21 can input voice commands corresponding to the displayed badges. For example, if the user 21 inputs "Open the first one," the large-screen device 22 will process the voice command after receiving it, obtain the corresponding voice instruction, and match the voice instruction with the unique identifier of each control. If the voice instruction matches badge 235, the large-screen device 22 will determine that control 224 is the control the user actually wants to control and will then perform a click operation on control 224. Similarly, if the user inputs "Open the second one," and badge 236 matches, then a click operation will be performed on control 225.

[0077] To further enhance the user experience, after displaying the unique identifier, the large-screen device 22 can prompt the user to re-enter the control voice input via a prompt message. The prompt can be either a voice message or a text message. For example, such as... Figure 5 As shown, after displaying badges 235 and 236, the large-screen device 22 displays a prompt window 237. The prompt window 237 displays the prompt message "Please select the [number], you can say 'number 1'", that is, if the user wants to select control 224, they can say "number 1", and if the user wants to select control 225, they can say "number 2".

[0078] It should be noted that the scenario of controlling the large screen device 22 via voice mentioned above is merely an example and does not limit the application scenarios of the embodiments of this application.

[0079] In addition to introducing the possible application scenarios involved in the embodiments of this application, the solutions provided by the embodiments of this application will be described in illustrative terms below with reference to the accompanying drawings.

[0080] See Figure 6 This is a flowchart illustrating a voice interaction method provided in an embodiment of this application. The method may include the following steps:

[0081] Step S601: The electronic device acquires the first voice.

[0082] For example, an electronic device can acquire a user's voice through a microphone to obtain a first voice recording. The microphone may or may not be integrated into the electronic device. Specifically, the microphone may be a microphone array.

[0083] Step S602: The electronic device processes the first voice to obtain the first voice command.

[0084] For example, the electronic device performs ASR processing on the first voice to convert the first voice into a text control command to obtain the aforementioned first voice instruction.

[0085] Step S603: The electronic device obtains the text description information and position information of each first control in the current interface. The first control is a control belonging to the first category. The current interface displayed on the screen of the electronic device includes at least one control.

[0086] For example, the first category mentioned above can be system native controls. For system native controls, the electronic device can obtain the text description information and position information of each first control by traversing the interface layout file of the current interface.

[0087] The controls displayed on the current interface may include only native system controls, or they may include both native system controls and third-party custom controls and WebView controls, etc.

[0088] Step S604: The electronic device determines the visual focus area of ​​the human eye's line of sight on the screen.

[0089] In practical applications, electronic devices can use eye-tracking technology to determine the visual focus area of ​​a user's gaze on the screen.

[0090] For example, electronic devices capture user facial images through cameras, detect the facial region in the facial image, locate the pupil center based on the facial region, and finally determine the viewpoint coordinates of the pupil center on the screen according to the correspondence between the image coordinate system and the screen coordinate system. That is, by tracking the gaze, the visual focus area of ​​the human eye on the screen is obtained.

[0091] Step S605: The electronic device matches the first voice command with the target information, which includes the text description information of each first control.

[0092] Step S606: If the current interface includes at least two first target controls, the electronic device determines whether the first target control is located within the visual focus area based on the position information of the first target control for each first target control. The first target control is the control whose text description information matches the first voice command.

[0093] In practical applications, electronic devices can determine whether a first target control is located within the visual focus area based on the coordinates of each first target control and the coordinates of the visual focus area, and count the number of first target controls located within the visual focus area.

[0094] In other embodiments, if the current interface includes only one first target control, then a corresponding operation is performed on the first target control, such as performing a simulated click or double-click operation on the first target control.

[0095] If the current screen does not include the first target control, the conversation can be ended or the audio can continue to be received.

[0096] Step S607: If the visual focus area contains only one first target control, the electronic device performs a preset operation on the first target control within the visual focus area.

[0097] The preset operations mentioned above can be exemplified as single click, double click, or touch, etc. No limitation is imposed on the preset operations here.

[0098] It can be seen that if there are at least two primary target controls on the current interface displayed by the electronic device, i.e., a control matching conflict has occurred, the visual focus area of ​​the human eye on the screen is used for filtering to reduce the control matching range, reduce the possibility of control matching conflicts, and improve the control matching accuracy.

[0099] On the other hand, when a user inputs voice commands into an electronic device to control a specific control, their gaze is typically focused on the area containing that control. Therefore, if there are at least two primary target controls on the current interface, filtering these controls using a visual focus area can ensure that the matched controls better match the user's actual intent.

[0100] In the above embodiments, if the visual focus area includes only one first target control, then the first target control is taken as the control that the user actually wants to operate, and a preset operation is performed on the first target control.

[0101] If the visual focus area includes at least two primary target controls, the electronic device cannot directly perform corresponding operations on both of these controls. In this case, to further determine the user's intent, a unique identifier can be displayed for each primary target control within the visual focus area. The user can then input the corresponding voice command for each unique identifier to select one of the at least two primary target controls within the visual focus area.

[0102] See Figure 7 This is another flowchart illustrating the voice interaction method provided in this application embodiment. The method may include the following steps:

[0103] Step S701: The electronic device acquires the first voice.

[0104] Step S702: The electronic device processes the first voice to obtain the first voice command.

[0105] Step S703: The electronic device obtains the text description information and position information of each first control in the current interface. The first control is a control belonging to the first category. The current interface displayed on the screen of the electronic device includes at least one control.

[0106] Step S704: The electronic device determines the visual focus area of ​​the human eye's line of sight on the screen.

[0107] Step S705: The electronic device matches the first voice command with the target information, which includes the text description information of each first control.

[0108] Step S706: If the current interface includes at least two first target controls, the electronic device determines whether the first target control is located within the visual focus area based on the position information of the first target control for each first target control. The first target control is the control whose text description information matches the first voice command.

[0109] Step S707: If the visual focus area contains only one first target control, the electronic device performs a preset operation on the first target control within the visual focus area.

[0110] It is understood that the relevant explanations for steps S701 to S707 can be found above, and will not be repeated here.

[0111] Step S708: If the visual focus area includes at least two first target controls, the electronic device displays the unique identifier of each first target control within the visual focus area.

[0112] The specific form of the aforementioned unique identifier can be arbitrary and is not limited here.

[0113] For example, the aforementioned unique identifier can be specifically represented as follows: Figure 4 The superscript is a superscript added to each primary target control within the visual focus area. This superscript can be a number, a letter, or a symbol; there are no restrictions on its type.

[0114] In other embodiments, to further enhance the user experience, the electronic device may display a prompt message after displaying the unique identifier, or simultaneously with displaying the unique identifier. This prompt message is for the user to input voice input regarding the unique identifier. The specific form of the prompt message can be arbitrary. For example, the prompt message can be delivered to the user via voice prompts or via text prompts.

[0115] For example, see Figure 5 The large screen device 22 pops up a prompt window 237, which displays the message "Please select the number, you can say the first one".

[0116] Step S709: The electronic device acquires the second voice.

[0117] For example, when the unique identifier is as follows Figure 4 When the indicated badge is displayed, the second voice prompt will be "the first one" or "open the first one" when the user wants to open control 224.

[0118] Understandably, electronic devices can collect user voice data through microphones.

[0119] Step S710: The electronic device processes the second voice to obtain the second voice command.

[0120] Specifically, the electronic device can perform ASR processing on the second voice to obtain text control commands, and then obtain the second voice instructions.

[0121] Step S711: The electronic device matches the second voice command with the unique identifier of each first target control.

[0122] For example, see Figure 4 In the scenario shown, if the second voice command is "Open the first one", the large screen device 22 will match the "1" in the second voice command with subscripts 235 and 236 respectively; since subscript 235 is specifically the number 1, subscript 235 matches the "1" in the second voice command. At this time, the second target control is control 224.

[0123] Step S712: When there is only one second target control in the visual focus area, the electronic device performs a preset operation on the second target control. The second target control is the first target control whose unique identifier matches the second voice command.

[0124] It is understandable that each first target control within the visual focus area has a unique identifier, and the second voice input is based on the unique identifier. Therefore, under normal circumstances, there is only one first target control that matches the second voice command, and there is no need to count the number of second target controls.

[0125] In addition, when there is no second target control in the visual focus area, that is, when there is no first target control in the visual focus area that matches the second voice command, the electronic device can end the current dialogue or continue to receive audio, or prompt the user to input voice again.

[0126] It should be noted that after the electronic device matches the second voice command with the unique identifier, it then disables the display of the unique identifier for each first target control. For example, when the unique identifier is as follows... Figure 4 When the numeric badge is displayed, the numeric badge display will be turned off after the voice command and the numeric badge are matched.

[0127] It can be seen that if there are at least two primary target controls on the current interface displayed by the electronic device, i.e., a control matching conflict has occurred, the visual focus area of ​​the human eye on the screen is used for filtering to reduce the control matching range, reduce the possibility of control matching conflicts, and improve the control matching accuracy.

[0128] Furthermore, when there are at least two primary target controls in the visual focus area, a unique identifier is displayed for each primary target control in the visual focus area to reconfirm the user's control intent, thereby further improving the accuracy of control matching.

[0129] In addition, compared to displaying badges for every control in the entire interface, this embodiment only displays the unique identifier of each first target control in the visual focus area when there are matching conflicting controls, resulting in a better user interaction experience.

[0130] In the above embodiments, the electronic device matches the first voice command with target information, which includes text descriptions of each first control. In other embodiments, the target information may also include text descriptions of each second control within the visual focus area.

[0131] The second control belongs to the second category. This differs from the first category of controls, which are controls whose corresponding text descriptions can be obtained by traversing the interface layout file; the second category of controls are controls whose corresponding text descriptions cannot be obtained by traversing the interface text. For example, the second category includes at least one of third-party custom controls and WebView controls.

[0132] If a second control is included in the visual focus area, the text descriptions of these controls cannot be obtained by traversing the interface layout file, resulting in low control recognition coverage and consequently low accuracy in subsequent control matching.

[0133] To further improve the control recognition coverage and thus the subsequent control matching accuracy, after determining the visual focus area and before matching the first voice command with the target information, the above embodiments may further include the following steps:

[0134] First, based on the position and type information of each control, determine whether the visual focus area includes a second control.

[0135] Specifically, the interface layout file includes control type information for each control on the current interface. Based on the control's position and type information, it is determined whether a second control is included within the visual focus area. If a second control is included within the visual focus area, OCR recognition is considered necessary; otherwise, if a second control is not included within the visual focus area, OCR recognition is considered unnecessary.

[0136] Then, when the visual focus area includes at least one second control, optical character recognition is performed on the visual focus area to obtain the optical character recognition result. The optical character recognition result includes the text description information of each second control in the visual focus area, and may also include the coordinate information of each second control in the visual focus area.

[0137] It can be seen that by using OCR to recognize the text description information of each second control within the visual focus area, the control recognition coverage is improved.

[0138] At this point, the target information includes the text descriptions of each first control and the text descriptions of each second control within the visual focus area. The first voice command is matched against the text descriptions of each first control and each second control. If no first target control exists on the current screen, the dialogue ends or audio reception continues; if only one first target control exists on the current screen, a preset operation is performed on that first target control; if at least two first target controls exist on the current screen, the visual focus area is used to filter between the at least two first target controls. The specific process can be found above and will not be repeated here.

[0139] In the above embodiments, the electronic device first determines the visual focus area of ​​the human eye on the screen, and then uses the visual focus area to filter at least two first target controls that have matching conflicts.

[0140] In some embodiments, the electronic device may perform the gaze area estimation process only once and determine the estimated visual area as the final user visual focus area. However, the accuracy of the visual focus area obtained in this way is relatively low.

[0141] In other embodiments, to improve the accuracy of visual tracking and further improve the accuracy of subsequent control matching, the electronic device can determine the final user visual focus area based on the first voice command, the text description information of the control, and at least two estimated visual areas.

[0142] For example, firstly, the electronic device obtains at least two candidate visual focus areas by performing at least two visual focus area estimations, where the candidate visual focus area is the visual focus area of ​​the human eye on the screen.

[0143] Then, the first voice command is matched with the target information to obtain a third target control, which is a control whose text description information matches the first voice command. In some cases, the second target control can be equivalent to the third target control.

[0144] Understandably, when the target information only includes the text description information of the first control, the first voice command is matched with the text description information of the first control; when the target information includes the text description information of the first control and the text description information of the second control, the first voice command is matched with the text description information of the first control and the text description information of the second control, respectively.

[0145] When there is at least one third target control, and none of the candidate visual focus areas include the third target control, the area formed by the leftmost boundary line, rightmost boundary line, topmost boundary line, and bottommost boundary line of at least two candidate visual focus areas is taken as the visual focus area.

[0146] Understandably, each potential focal point area is typically a frame-shaped area.

[0147] In practical applications, the leftmost boundary line can be determined based on the leftmost coordinates of at least two candidate visual focus areas. After determining the leftmost coordinates, the line segment passing through the leftmost coordinates and perpendicular to the X-axis is used as the leftmost boundary line. Similarly, the rightmost boundary line is determined based on the rightmost coordinates of at least two candidate visual focus areas. The uppermost and lowermost boundary lines are determined based on the uppermost and lowermost coordinates of at least two candidate visual focus areas, respectively.

[0148] For example, see Figure 8 The diagram illustrates one method for determining visual focus areas. The screen 82 of the electronic device 81 includes multiple candidate visual focus areas. Specifically, through gaze tracking, four visual focus area estimations are performed to obtain four candidate visual focus areas. The candidate visual focus area obtained from the first estimation is area 83, the second estimation is area 84, the third estimation is area 85, and the fourth estimation is area 86. At this point, there is no third target control.

[0149] Based on the coordinates of regions 82, 84, 85, and 86, the leftmost boundary line of these four regions can be determined as the left boundary line of region 82, i.e., line segment 87; the rightmost boundary line of these four regions can be determined as the right boundary line of region 86, i.e., line segment 88; the uppermost boundary line of these four regions can be determined as the upper boundary line of region 84, i.e., line segment 89; and the lowermost boundary line of these four regions can be determined as the lower boundary line of region 85, i.e., line segment 810. Figure 8 In the text, line segments 87, 88, 89, and 810 have all been thickened.

[0150] The area enclosed by the bolded line segments 87, 88, 89, and 810 is defined as the user's final visual focus area. Figure 8 Region 811 in the middle.

[0151] It is understandable that, based on the coordinates of the third target control and the coordinates of each candidate visual focus area, it can be determined whether the third target control is located within the candidate visual focus area.

[0152] When there is at least one third target control, and at least one third target control is not located in the intersection area of ​​at least two candidate visual focus areas, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of the at least two candidate visual focus areas is taken as the visual focus area, and the candidate visual focus area is the candidate visual focus area including the third target control.

[0153] For example, see Figure 9 Another schematic diagram illustrating the determination of visual focus areas is shown. The screen 92 of the electronic device 91 includes multiple candidate visual focus areas. Specifically, through gaze tracking, four visual focus area estimations are performed to obtain four candidate visual focus areas. Among them, the candidate visual focus area obtained from the first estimation is area 93, the candidate visual focus area obtained from the second estimation is area 94, the candidate visual focus area obtained from the third estimation is area 95, and the candidate visual focus area obtained from the fourth estimation is area 96.

[0154] At this point, at least one third target control exists on the current interface displayed on screen 92. Furthermore, for each third target control, based on its coordinates and the coordinates of the candidate visual focus area, it is determined whether the third target control falls within the intersection region. The intersection region refers to the intersection of at least two candidate visual focus areas.

[0155] like Figure 9 As shown, there is an intersection between regions 93 and 94, between regions 93 and 95, between regions 94 and 96, and between regions 96 and 95.

[0156] The third target control 912 is located within region 94, but not within the intersection of regions 94 and 93. The third target control 913 is located within region 96, but not within the intersection of regions 94 and 96.

[0157] Furthermore, since both regions 94 and 96 include a third target control, both regions 94 and 96 are target candidate visual focus areas.

[0158] Based on the coordinates of regions 94 and 96, the leftmost boundary line of these two regions can be determined as the left boundary line of region 94, i.e., line segment 97; the rightmost boundary line of these two regions can be determined as the right boundary line of region 96, i.e., line segment 98; the uppermost boundary line of these two regions can be determined as the upper boundary line of region 94, i.e., line segment 99; and the lowermost boundary line of these two regions can be determined as the lower boundary line of region 96, i.e., line segment 910. Figure 9 In the text, line segments 97, 98, 99, and 910 have all been thickened.

[0159] The area enclosed by the bolded line segments 97, 98, 99, and 910 is defined as the user's final visual focus area. Figure 9 Region 911 in the middle.

[0160] When there is at least one third target control, and at least one third target control is located in the intersection area of ​​at least two target candidate visual focus areas, the intersection area is taken as the visual focus area.

[0161] For example, see Figure 10 Another schematic diagram illustrating the determination of visual focus areas is shown. The screen 102 of the electronic device 101 includes multiple candidate visual focus areas. Specifically, through gaze tracking, four visual focus area estimations are performed to obtain four candidate visual focus areas. The candidate visual focus area obtained from the first estimation is area 103, the candidate visual focus area obtained from the second estimation is area 104, the candidate visual focus area obtained from the third estimation is area 105, and the candidate visual focus area obtained from the fourth estimation is area 106.

[0162] At this time, at least one third target control exists on the current interface displayed on screen 102. Since both areas 105 and 106 include the third target control 108, both areas 105 and 106 are target candidate visual focus areas. Furthermore, the third target control 108 is located in the intersection area between areas 105 and 106, so the intersection area between areas 105 and 106 is taken as the user's final visual focus area. Figure 10 Region 107 in the middle.

[0163] As can be seen, the embodiments of this application determine the user's final visual focus area by combining voice commands, text description information of controls, and multiple estimated visual areas, thereby improving the accuracy of eye tracking.

[0164] In the above embodiments, the electronic device first matches the first voice command with the target information. If at least two first target controls exist on the current interface, the visual focus area is used to filter the at least two first target controls that have matching conflicts. That is, the electronic device first performs a global match, and when the result of the global match is that there are at least two controls with matching conflicts, the visual focus area is used for further filtering.

[0165] In other embodiments, the electronic device may also skip global matching and instead directly match the first voice command with controls within the visual focus area.

[0166] In comparison, performing a global match first can prevent situations where the control the user actually wants to control is not located in the visual focus area, resulting in a higher control matching accuracy.

[0167] For example, see Figure 11 This is another schematic flowchart of a voice interaction method provided in an embodiment of this application. The method may include the following steps:

[0168] Step S1101: The electronic device acquires the first voice.

[0169] Step S1102: The electronic device processes the first voice to obtain the first voice command.

[0170] Step S1103: The electronic device determines the visual focus area of ​​the human eye's line of sight on the screen of the electronic device, and the current interface displayed on the screen includes at least one control.

[0171] Step S1104: The electronic device obtains the text description information of each first control, wherein the first control is a control that belongs to the first category and is located in the visual focus area.

[0172] Step S1105: The electronic device matches the first voice command with the target information, which includes the text description information of each first control.

[0173] Step S1106: When there is only one first target control in the visual focus area, the electronic device performs a preset operation on the first target control. The first target control is a control whose text description information matches the first voice command.

[0174] It should be noted that the same or similar aspects of this embodiment and the above embodiments can be found above, and will not be repeated here.

[0175] As can be seen, the embodiments of this application match the first voice command with the controls within the visual focus area, which can reduce the control matching range, reduce the possibility of control matching conflicts, and thus improve the control matching accuracy.

[0176] In other embodiments, when there are at least two first target controls within the visual focus area, the electronic device can display a corresponding unique identifier for each first target control, acquire the user's second voice command in response to the unique identifier, and then match the voice command corresponding to the second voice command with the unique identifier, executing the corresponding operation based on the matching result. This improves the user interaction experience and increases the accuracy of control matching.

[0177] Of course, electronic devices can also display prompts to users, prompting them to input voice for a unique identifier.

[0178] In other embodiments, the target information may further include text description information of the second control. The second control is information belonging to a second category. For example, the electronic device can determine whether the visual focus area includes third-party custom controls and WebView controls based on the control type information. If so, OCR recognition is performed on the visual focus area to obtain the OCR recognition result, and then the first voice command is matched with the OCR recognition result and the text description information of the first control, respectively. This improves the control recognition coverage and further enhances the control matching accuracy.

[0179] In other embodiments, when determining the visual focus area, the candidate visual focus area can be obtained by combining the text description of the control, voice commands, and multiple estimations to further improve the accuracy of visual tracking. Please refer to the above for details, which will not be repeated here.

[0180] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0181] This application also provides a voice interaction device. See [link to relevant documentation]. Figure 12 This is a schematic block diagram of a voice interaction device provided in an embodiment of this application. The voice interaction device may include:

[0182] The data acquisition module 121 is used to acquire facial images and voice information. Specifically, it acquires facial images through a camera and acquires control voice through a sound recording system.

[0183] The eye-tracking module 122 is used for screen eye tracking based on facial images.

[0184] Specifically, the gaze tracking module 122 performs face region inspection on the face image, determines the face region in the face image, and determines the eye region from the face region; it performs pupil localization based on the eye region, and then performs screen gaze tracking based on the pupil center and the correspondence between the image coordinate system and the screen coordinate system to determine the visual focus area of ​​the human eye's gaze on the screen.

[0185] The control recognition module 123 is used to recognize the text description information and position information of the control.

[0186] Specifically, the control recognition module 123 can identify controls by traversing the interface layout file to obtain the text description information and coordinate information of the controls; it can also obtain the text description information of the controls through OCR recognition. For example, it can perform OCR recognition on the visual focus area to obtain the text description information and coordinate information of third-party custom controls or WebView controls within the visual focus area.

[0187] The control matching module 124 is used for ASR semantic recognition, control matching, and unique identifier matching.

[0188] Specifically, the control matching module 124 is used to perform ASR processing on the collected user voice, obtain the voice command corresponding to the user voice, and match the voice command corresponding to the user voice with the text description information of the control; it is also used to match the voice command corresponding to the user voice with the unique identifier of the control.

[0189] The interactive execution module 125 is used to perform preset operations on the control.

[0190] Specifically, the interaction execution module 125 performs preset operations such as clicking or double-clicking on the matched controls based on the matching results of the control matching module 124.

[0191] A flowchart based on a voice interaction device can be shown as follows: Figure 13 As shown. Figure 13 As shown, the data acquisition module can acquire facial image signals through a camera or infrared camera, and acquire voice information through a microphone array. The camera can be a monocular camera or a binocular camera.

[0192] After acquiring the voice signal, the control matching module can perform ASR processing on the voice signal to obtain voice commands.

[0193] After acquiring the facial image signal, the gaze tracking module sequentially performs face detection, pupil localization, and screen gaze tracking to determine the focal area of ​​the human eye's gaze on the screen. Furthermore, to improve gaze tracking accuracy, the module can determine the focal area based on control text descriptions, voice commands, and multiple estimated areas obtained through screen gaze tracking.

[0194] The control recognition module can obtain a list of controls by traversing the interface layout file, which includes information about the system's native controls; it can also obtain OCR text by performing regional OCR recognition on the visual focus area.

[0195] The control matching module matches the voice command with the control list and OCR text to perform control matching, and after obtaining the matching results, performs click operations on the controls based on the matching results.

[0196] Optionally, between the control being matched and the control click operation being performed, steps such as adding a unique identifier such as a numeric badge to the control, and conducting multi-turn dialogues based on the unique identifier may also be included.

[0197] It should be noted that the voice interaction scheme of this application embodiment can be divided into three parts: an image or voice signal acquisition part, a gaze tracking, control recognition, and natural speech recognition part, and a control matching and execution part. Among them, voice acquisition and voice ASR processing, control traversal, and gaze tracking can be performed simultaneously, and the execution order of these processes is not limited here.

[0198] Figure 14 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Figure 14 As shown, the electronic device 14 of this embodiment includes: at least one processor 140 ( Figure 14 The diagram shows only one processor, a memory 141, and a computer program 142 stored in the memory 141 and executable on the at least one processor 140, wherein the processor 140 executes the computer program 142 to implement the steps in any of the above-described voice interaction method embodiments.

[0199] The electronic device may include, but is not limited to, a processor 140 and a memory 141. Those skilled in the art will understand that... Figure 14 This is merely an example of electronic device 14 and does not constitute a limitation on electronic device 14. It may include more or fewer components than shown, or combine certain components, or different components, such as input / output devices, network access devices, etc.

[0200] The processor 140 may be a Central Processing Unit (CPU), or it may be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor.

[0201] In some embodiments, the memory 141 may be an internal storage unit of the electronic device 14, such as a hard disk or memory of the electronic device 14. In other embodiments, the memory 141 may be an external storage device of the electronic device 14, such as a plug-in hard disk, smart media card (SMC), secure digital (SD) card, flash card, etc., equipped on the electronic device 14. Furthermore, the memory 141 may include both internal and external storage units of the electronic device 14. The memory 141 is used to store the operating system, applications, bootloader, data, and other programs, such as the program code of the computer program. The memory 141 can also be used to temporarily store data that has been output or will be output.

[0202] Those skilled in the art will clearly understand that, for the sake of convenience and brevity, the above-described division of functional units and modules is merely an example. In practical applications, the above functions can be assigned to different functional units and modules as needed, that is, the internal structure of the device can be divided into different functional units or modules to complete all or part of the functions described above. The functional units and modules in the embodiments can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit. Furthermore, the specific names of the functional units and modules are only for easy differentiation and are not intended to limit the scope of protection of this application. The specific working process of the units and modules in the above system can be referred to the corresponding process in the foregoing method embodiments, and will not be repeated here.

[0203] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, all or part of the processes in the methods of the above embodiments of this application can be implemented by a computer program instructing related hardware. The computer program can be stored in a computer-readable storage medium, and when executed by a processor, it can implement the steps of the various method embodiments described above. The computer program includes computer program code, which can be in the form of source code, object code, executable files, or certain intermediate forms. The computer-readable medium can include at least: any entity or device capable of carrying computer program code to a photographing device / terminal device, a recording medium, a computer memory, a read-only memory (ROM), a random access memory (RAM), an electrical carrier signal, a telecommunication signal, and a software distribution medium. Examples include USB flash drives, portable hard drives, magnetic disks, or optical disks. In some jurisdictions, according to legislation and patent practice, computer-readable media cannot be electrical carrier signals or telecommunication signals.

[0204] This application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, implements the steps described in the various method embodiments above.

[0205] This application provides a computer program product that, when run on an electronic device, enables the electronic device to perform the steps described in the various method embodiments above.

[0206] This application also provides a chip system, which includes a processor coupled to a memory. The processor executes a computer program stored in the memory to implement the methods described in the above embodiments. The chip system may be a single chip or a chip module composed of multiple chips.

[0207] In the above embodiments, the descriptions of each embodiment have their own emphasis. Parts not detailed or described in a particular embodiment can be referred to in the relevant descriptions of other embodiments. It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application. Furthermore, in the description of this application specification and the appended claims, the terms "first," "second," "third," etc., are only used to distinguish descriptions and should not be construed as indicating or implying relative importance. References to "one embodiment" or "some embodiments" in this application specification mean that one or more embodiments of this application include a specific feature, structure, or characteristic described in connection with that embodiment. Therefore, the phrases "in one embodiment," "in some embodiments," "in other embodiments," "in yet other embodiments," etc., appearing in different parts of this specification do not necessarily refer to the same embodiment, but rather mean "one or more, but not all, embodiments," unless otherwise specifically emphasized.

[0208] Finally, it should be noted that the above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any variations or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A voice interaction method, characterized in that, Applied to electronic devices, the method includes: Get the first audio recording; The first voice is processed to obtain a first voice command; Obtain the text description and position information of each first control within the current interface, wherein the first control is a control belonging to a first category, and the current interface displayed on the screen of the electronic device includes at least one control; the controls of the first category include system native controls; Determine the visual focal area of ​​the human eye on the screen; The first voice command is matched with the target information to determine the first target control, wherein the target information includes the text description information of each of the first controls; When the current interface includes at least two first target controls, for each first target control, it is determined whether the first target control is located within the visual focus area based on the position information of the first target control. The first target control is a control whose text description information matches the first voice command. When the visual focus area includes only one first target control, a preset operation is performed on the first target control within the visual focus area.

2. The method according to claim 1, characterized in that, When the visual focus area includes at least two of the first target controls; the method further includes: Display a unique identifier for each of the first target controls within the visual focus area; Obtain the second voice; The second voice is processed to obtain a second voice command; The second voice command is matched with the unique identifier of each of the first target controls; When there is only one second target control in the visual focus area, the preset operation is performed on the second target control. The second target control is the first target control that uniquely identifies the second voice command.

3. The method according to claim 2, characterized in that, After displaying the unique identifier of each of the first target controls within the visual focus area and before acquiring the second voice, the method further includes: A prompt message is displayed, which prompts the user to input voice for the unique identifier.

4. The method according to claim 1, characterized in that, The target information also includes text description information of each second control within the visual focus area, wherein the second control is a control belonging to a second category; Before matching the first voice command with the target information, the method further includes: Traverse the page layout file of the current interface to obtain the position information and control type information of each control; Based on the position information and control type information of each control, determine whether the second control is included in the visual focus area; When the visual focus area includes at least one of the second controls, optical character recognition is performed on the visual focus area to obtain an optical character recognition result, which includes text description information of each of the second controls within the visual focus area.

5. The method according to claim 4, characterized in that, The second category of controls includes WebView controls and / or third-party custom controls.

6. The method according to any one of claims 1 to 5, characterized in that, Determining the visual focal area of ​​the human eye on the screen includes: By performing at least two visual focus area estimations, at least two candidate visual focus areas are obtained, wherein the candidate visual focus area is the visual focus area of ​​the human eye on the screen. Match the first voice command with the text description information of each of the first controls; When there is at least one third target control, and none of the candidate visual focus areas include the third target control, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of the at least two candidate visual focus areas is taken as the visual focus area, and the third target control is the first control whose text description information matches the first voice command. When there is at least one third target control, and at least one third target control is not located in the intersection area of ​​at least two candidate visual focus areas, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of at least two candidate visual focus areas is taken as the visual focus area, and the candidate visual focus area is the candidate visual focus area including the third target control. When there is at least one third target control, and at least one third target control is located in the intersection area of ​​at least two target candidate visual focus areas, the intersection area is taken as the visual focus area.

7. The method according to claim 1, characterized in that, When the current interface includes only one of the first target controls, the method further includes: Perform the preset operation on the first target control.

8. A voice interaction method, characterized in that, Applied to electronic devices, the method includes: Get the first audio recording; The first voice is processed to obtain a first voice command; Determine the visual focus area of ​​the human eye on the screen of the electronic device, wherein the current interface displayed on the screen includes at least one control; Obtain the text description information of each first control, wherein the first control is a control belonging to the first category and located within the visual focus area; The first voice command is matched with target information, which includes text description information for each of the first controls; When there is only one first target control within the visual focus area, a preset operation is performed on the first target control, wherein the first target control is a control whose text description information matches the first voice command; Determining the visual focal area of ​​the human eye on the screen of the electronic device includes: By performing at least two visual focus area estimations, at least two candidate visual focus areas are obtained, wherein the candidate visual focus area is the visual focus area of ​​the human eye on the screen. Match the first voice command with the text description information of each of the first controls; When there is at least one third target control, and none of the candidate visual focus areas include the third target control, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of the at least two candidate visual focus areas is taken as the visual focus area, and the third target control is the first control whose text description information matches the first voice command. When there is at least one third target control, and at least one third target control is not located in the intersection area of ​​at least two candidate visual focus areas, the area formed by the leftmost boundary line, the rightmost boundary line, the topmost boundary line and the bottommost boundary line of at least two candidate visual focus areas is taken as the visual focus area, and the candidate visual focus area is the candidate visual focus area including the third target control. When there is at least one third target control, and at least one third target control is located in the intersection area of ​​at least two target candidate visual focus areas, the intersection area is taken as the visual focus area.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the computer program, it implements the method as described in any one of claims 1 to 7 or 8.

10. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method as described in any one of claims 1 to 7 or 8 to 9.

11. A computer program product, characterized in that, When the computer program product is run on an electronic device, it causes the electronic device to perform the method as described in any one of claims 1 to 7 or 8.

Images