Input interaction method, electronic equipment and readable medium

Abstract

The invention provides an input interaction method, electronic equipment and a readable medium, in the process that the electronic equipment displays a first interface, when a user lifts the electronic equipment to speak beside the mouth, a first voice-to-text function is started, and the electronic equipment displays a text corresponding to the voice input by the user in a first text input area. A user views application scenes such as notices and input texts, an interface displayed by the electronic equipment comprises the first text input area, the user can lift the electronic equipment to speak beside the mouth, the electronic equipment starts the first voice-to-text function, the user does not need to additionally operate a screen of the electronic equipment, and the user experience is improved. The electronic equipment can display the text corresponding to the voice input by the user in the first text input area, the operation of the user on a screen is simplified, and convenience is brought to the user.

Description

Technical Field

[0001] The present application relates to the field of data processing technology, and in particular to an input interaction method, electronic device, computer program product, and computer-readable storage medium. Background Art

[0002] When users hold electronic devices such as mobile phones in one hand, they usually use their thumbs to input and click on the display screen. However, the thumb's operating range is difficult to cover both the top and bottom of the device at the same time.

[0003] In application scenarios such as when users view notifications and input text, the user's operations involve the top and bottom of the device. If the user uses the electronic device by holding the device with one hand, the thumb usually cannot reach both the top and bottom of the device, which brings great inconvenience to the user's operation. Summary of the Invention

[0004] The present application provides an input interaction method, electronic device, computer program product and computer-readable storage medium, the purpose of which is to simplify user operations on the screen in application scenarios such as notifications and text input, thereby bringing convenience to users.

[0005] In order to achieve the above objectives, this application provides the following technical solutions:

[0006] In a first aspect, the present application provides an input interaction method, including: an electronic device displays a first interface, the first interface includes a first text input area; during the process of the electronic device displaying the first interface, when the user lifts the electronic device to his mouth to speak, a first voice-to-text function is turned on, and the electronic device displays the text corresponding to the user input voice in the first text input area; in response to the user's operation on the first text input area, the electronic device displays a second interface, the second interface includes: a voice-to-text button and a second text input area; during the process of the electronic device displaying the second interface, when the user presses the voice-to-text button, the second voice-to-text function is turned on, and when the user input voice is detected, the text corresponding to the user input voice is displayed in the second text input area.

[0007] It can be seen from the above content that: in application scenarios such as users viewing notifications and entering text, the interface displayed by the electronic device includes a first text input area. The user can lift the electronic device and put it to his mouth to speak. The electronic device turns on the first voice-to-text function. The user does not need to operate the screen of the electronic device additionally. The electronic device can display the text corresponding to the voice input by the user in the first text input area, which simplifies the user's operation of the screen and brings convenience to the user.

[0008] In some embodiments, the first speech-to-text function is a breath-activated speech-to-text function, and the second speech-to-text function is a speech-to-text function activated by the user pressing a speech-to-text button. The first speech-to-text function has stronger noise immunity than the second speech-to-text function, and can largely block out interference signals during the user's speech input.

[0009] In one possible implementation, when the user lifts the electronic device and puts it to his mouth to speak, the user does not operate the electronic device.

[0010] In a possible implementation, the first interface does not include a speech-to-text button, or the first interface does not include an interface of an input method application.

[0011] In a possible implementation, the first interface also includes an interface of a first application.

[0012] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays a third interface, the third interface includes a first banner notification of the first application; wherein the electronic device displays the first interface, including: displaying the first interface in response to a click operation of the user's finger on the first banner notification.

[0013] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays a third interface, the third interface includes a first banner notification of the first application; the electronic device displays the first interface, including: in the process of the electronic device displaying the third interface, in response to the user looking at the first banner notification, the electronic device displays the first interface.

[0014] In this possible implementation, the electronic device displays a third interface, which includes a first banner notification of the first application. The user does not need to operate the screen of the electronic device but only looks at the first banner notification. The electronic device can then display the first interface, further facilitating user operation.

[0015] In one possible implementation, in response to a user operation on a text input area of ​​a first interface, the electronic device displays a second interface, including: in response to a user click operation on the first text input area, the electronic device displays the second interface.

[0016] In one possible implementation, it also includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to his mouth to speak, the electronic device wakes up the voice assistant, wherein the fourth interface does not include a text input area. In this possible implementation, the fourth interface may be an interface of the electronic device that is different from the first interface. In some embodiments, the fourth interface does not include an interface for an input method application. When the electronic device displays the fourth interface or the electronic device is in the screen-off state, the user lifts the electronic device to his mouth to speak to wake up the voice assistant, and the voice assistant responds to the instructions corresponding to the user's input voice. In some embodiments, when the user lifts the electronic device to his mouth to speak, the electronic device waking up the voice assistant can be called a breath-awakening voice assistant function.

[0017] In one possible embodiment, the electronic device is provided with a first switch, and the first switch is in an on state. In this possible embodiment, the first switch corresponds to a first speech-to-text function. When the switch is in the on state, the first speech-to-text function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0018] In one possible implementation, the first switch is set on the interface of the first setting item of the input method application; wherein, when the first switch is turned on, the interface of the first setting item displays a first pop-up window, and the first pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0019] In a possible implementation, the electronic device is configured with a first switch setting item on the interface of the input method application, and the interface of the first switch setting item includes a button of the first switch and a guide animation.

[0020] In one possible embodiment, the first switch is a speech-to-text button of an input method application. The duration of the speech-to-text button being pressed for the first time is greater than a threshold. The electronic device displays a second pop-up window. The second pop-up window includes a first text control. The first text control is clicked, the function corresponding to the first switch is turned on, and a third pop-up window is displayed. The third pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0021] In one possible implementation, the electronic device is provided with a first switch, and the first switch is in an off state. The method further includes: while the electronic device is displaying the first interface, when the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function. In this possible implementation, when the first switch is in the off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function, and the first text input area of ​​the electronic device does not display text, thereby disabling the first speech-to-text function.

[0022] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an on state. In this possible embodiment, the second switch is a switch for a breath-activated voice assistant function. When the switch is in the on state, the breath-activated voice assistant function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0023] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an off state. The method further includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to their mouth to speak, the electronic device does not awaken the voice assistant. In this possible embodiment, when the second switch is in an off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the breath-activated voice assistant function, and the voice assistant does not respond to commands corresponding to the user's voice input, thereby disabling the breath-activated voice assistant function.

[0024] In a second aspect, the present application provides an input interaction method, including: an electronic device displays a first interface, the first interface including a first text input area and a speech-to-text button; during the process of the electronic device displaying the first interface, when the user lifts the electronic device and puts it to his mouth to speak, the electronic device turns on the first speech-to-text function, switches the speech-to-text button from the first state to the second state, and displays the text corresponding to the user's voice input in the first text input area, wherein the user does not operate the speech-to-text button when the user lifts the electronic device and puts it to his mouth to speak.

[0025] It can be seen from the above content that: in application scenarios such as users viewing notifications and entering text, the interface displayed by the electronic device includes a first text input area and a voice-to-text button. The user can lift the electronic device and put it to his mouth to speak. The electronic device turns on the first voice-to-text function. The user does not need to operate the screen of the electronic device additionally. The electronic device can display the text corresponding to the voice input by the user in the first text input area, which simplifies the user's operation of the screen and brings convenience to the user.

[0026] In one possible embodiment, the first state is an off state, and the second state is a first on state. The method further includes: in the first on state, the electronic device continuously receives the user's voice input and converts the voice input into text and displays it in the first text input area. In this possible embodiment, the voice-to-text button is switched to the first on state, and the user lifts the electronic device to their mouth and continues to input voice. The first voice-to-text function is continuously activated, and the electronic device continuously receives the user's voice input and converts the voice input into text and displays it in the text input area. In this way, the electronic device converts the user's continuously input voice into text for display.

[0027] In one possible embodiment, the first state is the second on state, and the second state is the first on state. In the second on state, when the electronic device detects that the user clicks the voice-to-text button, it receives the user's voice input and converts the voice input into text and displays it in the first text input area; in the first on state, the electronic device continues to receive the user's voice input and converts the voice input into text and displays it in the first text input area.

[0028] In this possible implementation, the speech-to-text button includes a first on state and a second on state. In the first on state, when the user clicks the speech-to-text button, the electronic device can convert the speech input by the user into text and display it in the input text area. In the second on state, it indicates that the first speech-to-text function is activated. When the user lifts the electronic device and continues to input speech, the electronic device can convert the speech input by the user into text and display it in the first text input area.

[0029] In one possible implementation, the first text input area of ​​the first interface also displays text. In some embodiments, when the electronic device displays the first interface, the user lifts the electronic device to their mouth and speaks, and the first voice-to-text function is activated, and the electronic device displays the text corresponding to the user's input voice in the first text input area, or the user presses the voice-to-text button and activates the second voice-to-text function, and the electronic device displays the text corresponding to the user's input voice in the first text input area. In this way, the first text input area of ​​the first interface displayed by the electronic device also displays text. Moreover, when the user lifts the electronic device to their mouth and speaks again, the first voice-to-text function is activated, and the electronic device can continue to display the text corresponding to the user's input voice in the first text input area.

[0030] In a possible implementation, the first interface also includes an interface of a first application.

[0031] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes the first banner notification of the first application; in response to the user's click operation on the first banner notification, the third interface is displayed, the third interface includes the interface of the first application and the second text input area; the electronic device displays the first interface including: in response to the user's click operation on the second text input area, the first interface is displayed.

[0032] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; in response to the user's click operation on the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

[0033] In this possible implementation, the electronic device displays a first banner notification of the first application, which is usually located at the top of the electronic device. When the user clicks the first banner notification, the electronic device displays the third interface and then automatically displays the first interface. In this way, the user only needs to operate the top of the electronic device and does not need to operate the bottom of the electronic device, which brings great convenience to the user holding the electronic device with one hand.

[0034] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; the electronic device displays the first interface including: in the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the electronic device displays the first interface.

[0035] In this possible implementation, the electronic device displays a first banner notification of the first application. When the user looks at the first banner notification, the electronic device displays the first interface. The user does not need to operate the electronic device screen, thereby simplifying the user's operation of the screen.

[0036] In one possible implementation, when the user looks at the first banner notification, the user does not operate the screen of the electronic device.

[0037] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; in the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: in response to the user clicking the second text input area, the first interface is displayed.

[0038] In this possible implementation, the electronic device displays a first banner notification of the first application, which is usually located at the top of the electronic device. The user does not need to operate the top of the electronic device, but only needs to look at the first banner notification. The electronic device displays a third interface. The second text input area in the third interface is usually located at the bottom of the electronic device. The user clicks on the second text input area, and the electronic device displays the first interface. In this way, the user only needs to operate the bottom of the electronic device, and does not need to operate the top of the electronic device, which also brings great convenience to the scenario where the user holds the electronic device with one hand.

[0039] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; during the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

[0040] In this possible implementation, the electronic device displays the first banner notification of the first application. The user does not need to operate the top of the electronic device, but only needs to look at the first banner notification. The electronic device displays the third interface. After the electronic device displays the third interface, it automatically displays the first interface. In this way, the user does not need to operate the electronic device screen, and the operation is more convenient.

[0041] In one possible implementation, when the user looks at the first banner notification, the user does not operate the screen of the electronic device.

[0042] In one possible implementation, it also includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to his mouth to speak, the electronic device wakes up the voice assistant, wherein the fourth interface does not include a text input area. In this possible implementation, the fourth interface may be an interface of the electronic device that is different from the first interface. In some embodiments, the fourth interface does not include an interface for an input method application. When the electronic device displays the fourth interface or the electronic device is in the screen-off state, the user lifts the electronic device to his mouth to speak to wake up the voice assistant, and the voice assistant responds to the instructions corresponding to the user's input voice. In some embodiments, when the user lifts the electronic device to his mouth to speak, the electronic device waking up the voice assistant can be called a breath-awakening voice assistant function.

[0043] In one possible embodiment, the electronic device is provided with a first switch, and the first switch is in an on state. In this possible embodiment, the first switch corresponds to a first speech-to-text function. When the switch is in the on state, the first speech-to-text function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0044] In one possible implementation, the first switch is set on the interface of the first setting item of the input method application; wherein, when the first switch is turned on, the interface of the first setting item displays a first pop-up window, and the first pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0045] In a possible implementation, the electronic device is configured with a first switch setting item on the interface of the input method application, and the interface of the first switch setting item includes a button of the first switch and a guide animation.

[0046] In one possible embodiment, the first switch is a speech-to-text button of an input method application. The duration of the speech-to-text button being pressed for the first time is greater than a threshold. The electronic device displays a second pop-up window. The second pop-up window includes a first text control. The first text control is clicked, the function corresponding to the first switch is turned on, and a third pop-up window is displayed. The third pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0047] In one possible implementation, the electronic device is provided with a first switch, and the first switch is in an off state. The method further includes: while the electronic device is displaying the first interface, when the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function. In this possible implementation, when the first switch is in the off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function, and the first text input area of ​​the electronic device does not display text, thereby disabling the first speech-to-text function.

[0048] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an on state. In this possible embodiment, the second switch is a switch for a breath-activated voice assistant function. When the switch is in the on state, the breath-activated voice assistant function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0049] In a third aspect, the present application provides an electronic device comprising: one or more processors, a memory, and a display screen; the memory and the display screen are coupled to the one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the input interaction method provided in any one of the first aspects, or the input interaction method provided in any one of the second aspects.

[0050] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the input interaction method provided in any one of the first aspects, or the input interaction method provided in any one of the second aspects.

[0051] In a fifth aspect, the present application provides a computer program product, which, when executed on a computer, enables the computer to execute the input interaction method provided in any one of the first aspects, or the input interaction method provided in any one of the second aspects. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] Figure 1 A diagram showing a user holding a mobile phone in one hand according to an embodiment of the present application;

[0053] Figure 2 An application scenario diagram of the input interaction method provided in an embodiment of the present application;

[0054] Figures 3A to 3E Several application scenarios of using the breath awakening function provided in the embodiments of this application;

[0055] Figures 4 to 7 This is a diagram showing the activation of the breath-awakened speech-to-text function provided in an embodiment of the present application;

[0056] Figure 8 A diagram showing another way to enter the interface of the input method provided in an embodiment of the present application;

[0057] Figure 9 A hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0058] Figure 10 A software framework diagram of an electronic device provided in an embodiment of the present application;

[0059] Figure 11 Schematic diagram of the input method and voice assistant registration service provided in the embodiment of this application;

[0060] Figure 12 A signaling diagram of the input interaction method provided in an embodiment of the present application;

[0061] Figure 13 A signaling diagram of another input interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0062] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0063] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0064] The "multiple" in the embodiments of the present application refers to greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0065] Before introducing the embodiments of this application, we first explain some of the terms or concepts involved in the embodiments of this application. It should be understood that this application does not specifically limit the naming of the following terms. The following terms may have other names. The renamed terms still meet the following relevant term explanations.

[0066] 1. Breath Wake-Up: An electronic device recognizes the characteristics of a user speaking close to a microphone. The electronic device can detect whether the user lifts the device and inputs a voice message to determine whether the user is close to the microphone. The breath wake-up feature can be used by multiple applications on the electronic device, which can use the results of the breath wake-up feature to perform tasks.

[0067] 2. Breath awakening voice assistant: The electronic device recognizes the above-mentioned breath awakening characteristics, thereby waking up the voice assistant to respond to the voice interaction command. This replaces the method of waking up the voice assistant when recognizing the user's call for special words in the related art, making it easier for users to use the voice assistant. After the electronic device recognizes the characteristics generated when the user speaks close to the microphone, the electronic device can automatically wake up the voice assistant, and the voice assistant (such as YOYO) will conduct voice interaction with the user. For example, the user only needs to lift the electronic device and say a voice interaction command (which may not be a special command to wake up the voice assistant), and the electronic device can respond to the voice interaction command, without the user having to say a voice command to wake up the voice assistant before saying the voice interaction command.

[0068] 3. Breath-activated speech-to-text: Electronic devices recognize the characteristics of breath-activated speech and activate the input method's speech-to-text service. In specific scenarios, the electronic device recognizes the characteristics of the user speaking close to the microphone and automatically activates the input method's speech-to-text function. For example, when the electronic device's input method is activated, the user lifts the electronic device and speaks. The electronic device then converts the user's voice input into text and displays it in the input method's text input box.

[0069] 4. Eye tracking function: It can be understood as the electronic device tracking the location of the user's gaze point, and by tracking the location of the gaze point, determining whether to expand the banner notification and / or open the application corresponding to the banner notification.

[0070] Figure 1 Shows the user holding the phone in one hand. Figure 1 As shown in (a), when holding the phone with one hand, the touch range of the thumb is area 101. The user holds the phone with one hand close to the top of the phone, and the touch range of the thumb is as follows Figure 1 As shown in (b), it is area 102. It can be seen from this that when a user holds the mobile phone in one hand, it is difficult for the thumb to touch both the top and the bottom of the mobile phone at the same time.

[0071] In an application scenario, such as Figure 2 As shown in (a), the top of the mobile phone displays a chat banner notification 201 of the instant messaging application. The mobile phone receives a click operation of the user on any position in the chat banner notification, and the mobile phone Figure 2 As shown in (b), the application interface corresponding to the chat banner notification is displayed. The application interface includes a switch button 202 and a text input box 203. The user clicks the switch button 202, and the mobile phone responds to the user's click operation and switches from text input to voice input. The text input box 203 is used to provide the user with text information. Figure 2 As shown in (b), click on the text input box 203, the mobile phone responds to the user's click operation, pulls up the input method and displays the following Figure 2 The interface shown in (c) of FIG. The interface includes an input method keyboard, which includes a speech-to-text button 204. When a user long-presses speech-to-text button 204, the electronic device's input method enters speech-to-text mode. The electronic device can collect the user's input speech and convert it into text, which can be displayed in text input box 203. Text display box 203 can be understood as the text display area of ​​the input method application.

[0072] Figure 2 In the illustrated application scenario, a chat banner notification 201 is located at the top of the phone, while a toggle button 202 and a speech-to-text button 204 are located at the bottom. If a user holds the phone in one hand and taps the chat banner notification 201 with their thumb, they won't be able to tap the toggle button 202 or the speech-to-text button 204, causing inconvenience to the user.

[0073] Based on this, an embodiment of the present application provides an input interaction method that can use breath awakening to automatically activate the voice input to text function, collect the user's input voice, and convert it into text without the user having to manually activate the voice input to text function.

[0074] Take mobile phones as an example. Figures 3A to 3E The application scenario of using the breath awakening function provided by the embodiment of the present application is demonstrated.

[0075] Figure 3A This shows a scenario where the input method uses breath wakeup to realize the voice input to text function, that is, the breath wakeup voice to text scenario.

[0076] like Figure 3A As shown in (a), the top of the mobile phone displays a chat banner notification 201 of the instant messaging application. The mobile phone receives a click operation of the user on any position in the chat banner notification, and the mobile phone Figure 3A As shown in (b), the application interface corresponding to the chat banner notification is displayed. Figure 3A As shown in (b), click on the text input box 203, the mobile phone responds to the user's click operation, pulls up the input method and displays the following Figure 3A The user lifts up the phone and inputs a voice. The phone's voice-to-text function is activated. The phone collects the user's input voice, converts the voice into the corresponding text, and then Figure 3A As shown in (e), the converted text is displayed in the text input box.

[0077] In another possible implementation, the user can click on the text input box 203 without clicking on the text input box 203. Figure 3A When the user lifts the phone and inputs a voice message, the phone's input method's voice-to-text function is activated. Figure 3AIn the interface shown in (e), the user's input voice is converted into text and displayed in the text input box; alternatively, the mobile phone can also display Figure 3A The interface shown in (c) is displayed again. Figure 3A In the interface shown in (e), the user's voice input is converted into text and displayed in the text input box.

[0078] In another possible implementation, if the user does not click on the text input box 203, it is understandable that the input method is not launched and the mobile phone displays Figure 3A When the interface shown in (b) is displayed, the user lifts the phone and inputs a voice, the breath-awakening voice assistant function of the phone is activated, and the voice assistant is enabled to run in response to the voice interaction instructions input by the user, and Figure 3A The upper layer of the interface shown in (b) shows text. For example, the following Figure 3E (b) shows a way to display text on top of an interface.

[0079] In another possible implementation, the user lifts the phone and inputs a voice, the breath-activated voice-to-text function of the phone is activated, the phone collects the user's input voice, converts the voice into corresponding text, and Figure 3A In (e), after the converted text is displayed in the text input box, the user is silent for a period of time. When the duration of this period reaches a threshold, the phone determines that the user is not inputting voice, and the breath-activated voice-to-text function ends. Furthermore, if the user inputs voice again after being silent for at least the threshold time, the phone will not convert the user's input voice into the corresponding text, and will not display the text in the text input box. In some embodiments, if the user lifts the phone again and inputs voice, the phone's breath-activated voice-to-text function is activated, and the phone collects the user's input voice, converts the voice into the corresponding text, and displays it in the text input box.

[0080] In another possible embodiment, after the breath-wake voice-to-text function of the mobile phone is activated, the mobile phone collects the voice input by the user and converts the voice into the corresponding text. In the process, the user inputs the operation of closing the input method application. For example, the user Figure 3A Click on the instant messaging application interface in (e), or the user Figure 3A The interface shown in (e) above enters the operation of exiting the instant messaging application, or the user Figure 3A In the middle (e), the user opens another app by typing, or closes the input method app by opening another app through an air gesture. In response to the user's action, the phone controls the input method app to exit, and the breath-activated speech-to-text feature ends. If the user lifts the phone and enters a voice input, the phone will not convert the user's voice input into text, nor will the text be displayed in the text input box.

[0081] Figure 3B Another scene showing breath-awakened speech-to-text conversion.

[0082] In some application scenarios, users Figure 3B As shown in (a), press the voice-to-text button 204 and input the voice. The mobile phone responds to the user's operation and starts the voice-to-text function. The input method converts the user's input voice into the corresponding text and displays it in the text input box. For example, Figure 3B After the user uses the speech-to-text button 204 to input a text, the user Figure 3B As shown in (c), lift the phone to your mouth and input voice, the phone activates the breath wake-up voice-to-text function, or the user Figure 3B As shown in (b), click on the interface of the instant messaging application, and the phone will display the following Figure 3B After the interface shown in (e) in the figure, the user can Figure 3B As shown in (c), lift the phone to your mouth and input voice, the phone activates the breath-awakened voice-to-text function. The input method converts the user's input voice into the corresponding text and Figure 3B As shown in (d), the converted text is displayed in the text input box.

[0083] In another possible implementation, the user clicks Figure 3B Click "Send" as shown in (b) and the phone will display Figure 3A In the interface shown in (c), of course, the area showing the user dialogue should also show "Hello" in the text input box. At this time, the user lifts the phone and inputs a voice, and the breath-wake-up voice-to-text function of the phone's input method is activated. The phone displays Figure 3A In the interface shown in (e), the user's voice input is converted into text and displayed in the text input box.

[0084] Figure 3C Another scene showing breath-awakened speech-to-text conversion.

[0085] In some application scenarios, it is difficult for users to click on the banner notification displayed at the top of the display. In this way, the mobile phone can use the eye tracking function to automatically open the application corresponding to the banner notification without the user performing a click operation. In some embodiments, the mobile phone receives a banner notification, activates the eye tracking function, and opens the application corresponding to the banner notification when it detects that the gaze point is in the area where the banner notification is located, or opens the application corresponding to the banner notification when it detects that the gaze point has been in the area where the banner notification is located for a preset time. When the mobile phone opens the application corresponding to the banner notification and displays the application interface, it directly pulls up the input method to run it, that is, the mobile phone detects that the user's gaze point has been in the area where the banner notification is located for a preset time, opens the application corresponding to the banner notification, and directly displays Figure 3C In the interface shown in (c) above, the lower half of the interface displays the content related to the input method. While the app corresponding to the banner notification is running, the user raises their phone and enters a voice message. The phone can also use the breath-activated speech-to-text feature to convert the user's voice message into text and display it in the text input box.

[0086] like Figure 3C As shown in (a) of FIG3C, a chat banner notification 201 of an instant messaging application is displayed on the top of the mobile phone, and the eye tracking function of the mobile phone is turned on. The chat banner notification 201 includes a gaze cursor 701, which indicates that the user's gaze point is located in the area where the banner notification is located. The mobile phone automatically enters the interface of the instant messaging application corresponding to the chat banner notification 201, that is, as shown in (b) of FIG3C, the interface of the instant messaging application is displayed. In some embodiments, the mobile phone Figure 3C In another embodiment, the mobile phone will not automatically pull up the input method to run. The user can Figure 3C As shown in (b), click on the text input box 203, the mobile phone responds to the user's click operation, pulls up the input method and displays the following Figure 3C The interface shown in (c) is shown in the figure.

[0087] The phone's input method is activated, the user lifts the phone, and inputs a voice message, such as Figure 3C As shown in (d), the breath-activated voice-to-text function of the phone is activated. The phone collects the user's input voice, converts the voice into the corresponding text, and Figure 3C As shown in (e), the converted text is displayed in the text input box.

[0088] In another possible implementation, the user can click on the text input box 203 without clicking on the text input box 203. Figure 3C When the user lifts the phone and inputs a voice message, the breath-activated voice-to-text function of the phone's input method is activated. Figure 3C In the interface shown in (e), the user's voice input is converted into text and displayed in the text input box.

[0089] Figure 3C In the application scenario shown, the phone automatically launches the input method or the user clicks the text input box to launch the input method. After the phone displays the interface shown in (c) in 3C, the user can also use Figure 3B In the display mode, by pressing the voice-to-text button 204 and starting the breath-awakening voice-to-text function, the input method converts the user's input voice into the corresponding text, and Figure 3B As shown in (b) and (d), the converted text is displayed in the text input box.

[0090] It should be noted that the chat banner notification 201 is displayed on the top of the mobile phone, and the chat banner notification 201 includes a gaze cursor 701. The mobile phone can also Figure 3D As shown in (a) and (b), the instant messaging application corresponding to the chat banner notification 201 is directly entered, and the input method is simultaneously launched and run.

[0091] In some embodiments, the phone detects a banner notification, activates eye tracking, and opens the app corresponding to the banner notification when it detects the gaze point is within the banner notification area. While the app corresponding to the banner notification is running, the phone launches an input method and converts the user's voice input into text, which is displayed in the input method's text input box without the user having to lift the phone.

[0092] It should also be noted that Figures 3A to 3C In the displayed scenario, compared with the voice-to-text function corresponding to the voice-to-text button, the mobile phone uses the breath-awakened voice-to-text function to convert the user's input voice into the corresponding text. It has strong anti-noise ability and can shield the interference signals in the user's voice input process to a large extent.

[0093] Figure 3E It shows a scenario where a voice assistant uses breath wake-up to respond to voice interaction commands, that is, a scenario where a voice assistant is woken up by breath.

[0094] like Figure 3E As shown in (a), the phone is in the off state. The user lifts the phone and enters a voice interaction command, such as a voice interaction command to query the weather. The phone can then start the voice assistant to respond to the voice interaction command, such as Figure 3D As shown in (b), the display screen is turned on to display the lock screen interface, and weather information is displayed on the lock screen interface.

[0095] Figures 3A to 3D The breath-activated speech-to-text function in the two application scenarios shown can be manually turned on or off by the user. Figures 4 to 8 Describes several ways to access the breath-awakening speech-to-text function.

[0096] In some embodiments, when a user first uses the voice-to-text function of an input method to input text, the user may be prompted to enable the breath-activated voice-to-text function. After the electronic device enables the breath-activated voice-to-text function in response to the user's operation, the input method of the mobile phone is launched again. The mobile phone can automatically activate the voice-to-text function of the input method using the breath-activated voice-to-text function to collect the user's input and convert it into text.

[0097] For example, Figure 4As shown in (a), the user long presses the voice-to-text button 204 for the first time in the interface of the instant messaging application. In response to the long press operation of the user, the mobile phone can display a pop-up window 205 to remind the user to turn on the breath-wake-up voice-to-text function. The pop-up window 205 can be displayed on the upper layer of the interface of the real-time messaging application. The pop-up window 205 can be a toast pop-up window. For example, Figure 4 As shown in (b), pop-up window 205 includes a prompt "Confirm to enable breath-activated speech-to-text function," a Cancel button, and an OK button. The user can click the Cancel button to ignore pop-up window 205, or click the OK button to enable breath-activated speech-to-text function. Clicking the Cancel button will cause pop-up window 205 to disappear, and the user can then manually operate speech-to-text button 204 to input speech.

[0098] User Figure 4 As shown in (b), click the OK button. In response to the user's click operation, the mobile phone displays a prompt window 206. The prompt window 206 can be displayed on the upper layer of the instant messaging application interface, and the prompt window 206 can be a toast pop-up window. Figure 4 As shown in (c), the prompt window 206 includes instructions for using the breath-activated speech-to-text function, such as "Reminder: Breath-activated speech-to-text is enabled. After positioning the cursor, aim at the microphone at the bottom of the device to start voice input." In this way, the breath-activated speech-to-text function is enabled.

[0099] For example, Figure 5 As shown in (a), the user long presses the voice-to-text button 204 for the first time in the note interface. In response to the user's click operation, the mobile phone can also display a pop-up window 205 to remind the user to turn on the breath-awakened voice-to-text function. The pop-up window 205 can be displayed on the upper layer of the note interface, for example, it can be displayed on the keyboard of the input method. The pop-up window 205 can also be a toast pop-up window. Figure 5 As shown in (b), the pop-up window 205 includes a prompt "Confirm to turn on the breath-awakened voice-to-text function", a cancel button and an confirm button.

[0100] User Figure 5 As shown in (b), click the OK button. In response to the user's click operation, the mobile phone displays a prompt window 206. The prompt window 206 can be displayed on the upper layer of the note interface, and the prompt window 206 can be a toast pop-up window. Figure 5 As shown in (c), the prompt window 206 includes instructions for using the breath-activated speech-to-text function, such as "Reminder: Breath-activated speech-to-text is enabled. After positioning the cursor, aim at the microphone at the bottom of the device to start voice input." In this way, the breath-activated speech-to-text function can also be enabled.

[0101] The "first time" mentioned in the above content can be understood as: the first long press of the voice-to-text button 204 in the following situations: 1. The new mobile phone includes the breath-awakened voice-to-text function of the input method, or 2. The software system of the old mobile phone is upgraded, and the upgraded system version includes the breath-awakened voice-to-text function of the input method.

[0102] In other embodiments, the user can also enable the breath-awakening voice-to-text function in the input method's settings interface. For example, the input setting item or interface setting item of the input method can set the breath-awakening voice-to-text function's enable control. Figure 6 and Figure 7 The setting interface of the input method is displayed, showing how to implement the control to enable the breath-awakened speech-to-text function.

[0103] Figure 6 It shows how to add the breath-awakened speech-to-text function to the input settings interface of the input method.

[0104] like Figure 6 As shown in (a), the phone displays the system and update interface of the settings application, and the system and update interface includes the language and input method item 501. It should be noted that the phone's settings application interface may include the system and update item, and the user can click the system and update item to display the following Figure 6 The system and updated interface shown in (a).

[0105] The user clicks on the language and input method item 501 on the system and update interface. The mobile phone responds to the user's click operation, such as Figure 6 As shown in (b), the language and input method interface is displayed. The language and input method interface includes XX input method item 502. The user clicks XX input method item 502. The mobile phone responds to the user's click operation, such as Figure 6 As shown in (c), the interface of the XX input method is displayed. The interface of the XX input method includes an input setting item 503. The user clicks the input setting item 503. The mobile phone responds to the user's click operation, such as Figure 6 As shown in (d), the input setting interface is displayed. The input setting interface includes a breath wake-up voice-to-text item 504. The breath wake-up voice-to-text item 504 has a sub-interface. When the user clicks the breath wake-up voice-to-text item 504, the mobile phone responds to the user's click operation, as shown in FIG. Figure 6 As shown in (e), the sub-interface for breath wake-up speech-to-text conversion is displayed. Exemplarily, this sub-interface includes a breath wake-up speech-to-text conversion guidance animation and a button 505. The breath wake-up speech-to-text conversion guidance animation can be understood as a short video that can explain the operation instructions of the breath wake-up function. The breath wake-up speech-to-text conversion button 505 can be provided to the user. The user can turn on or off the breath wake-up speech-to-text conversion function by clicking the breath wake-up speech-to-text conversion button 505.

[0106] Figure 7 Displays the button for starting the breath-activated speech-to-text function in the input method's input settings or interface settings.

[0107] Figure 7 (a) shows the interface of XX input method. Figure 7 (a) and Figure 6 (c) in the figure can be the same interface. The interface of the XX input method includes input setting item 503 and interface setting item 504. Figure 7 As shown in (a), the user clicks on the input setting item 503, and the mobile phone responds to the user's click operation and displays the input setting interface, such as Figure 7 As shown in (b), the input setting interface includes a breath awakening voice-to-text button 507, which allows the user to turn on or off the breath awakening voice-to-text function by clicking the button.

[0108] User Figure 7 As shown in (b), click the breath wake-up voice-to-text button 507, and the mobile phone responds to the user's click operation to turn on the breath wake-up voice-to-text function. In some embodiments, the mobile phone can also display a pop-up window to remind the user that the breath wake-up function is turned on. For example, Figure 7 (b) shows a pop-up window 508, which is displayed on the upper layer of the input settings interface and includes a reminder message "Breath wake-up speech-to-text is turned on." The pop-up window 508 can also be a toast pop-up window.

[0109] Or, as Figure 7 As shown in (a), the user clicks on the interface setting item 506, and the mobile phone responds to the user's click operation and displays the interface setting interface, such as Figure 7 As shown in (c), the interface setting interface includes a breath awakening voice-to-text button 509, which can be provided to the user to turn on or off the breath awakening voice-to-text function by clicking the button.

[0110] User Figure 7 As shown in (c), click the breath wake-up voice-to-text button 509, and the mobile phone responds to the user's click operation to turn on the breath wake-up voice-to-text function. In some embodiments, the mobile phone may also display a pop-up window to remind the user that the breath wake-up voice-to-text function is turned on. For example, Figure 7 In (c), a pop-up window 510 is shown, which is displayed on the upper layer of the interface setting interface and includes a reminder message "Breath wake-up speech-to-text is turned on." The pop-up window 510 can also be a toast pop-up window.

[0111] The input method setting interface is not limited to Figure 6 Shown entry method. Figure 8This shows another way to access the input method settings interface. After the input method is launched, the input method keyboard display interface includes an input method settings quick launch button, which the user can also use to enter the input method settings interface.

[0112] For example, Figure 8 The following shows the setting interface of launching input method in instant messaging application and note application. Figure 8 As shown in (a), the interface of the instant messaging application includes a quick start button 601 for setting the input method. When the user clicks the quick start button 601, the mobile phone displays buttons for multiple setting items of the input method on the interface of the instant messaging application in response to the user's click operation. Figure 8 As shown in (b), the more settings button 602 is usually an entry button for the input method settings interface. When the user clicks the more settings button 602, the mobile phone responds to the user's click operation and displays the following Figure 8 The setting interface of XX input method shown in (e).

[0113] Similarly, if Figure 8 As shown in (c), the interface of the note includes a quick start button 601 for setting the input method. When the user clicks the quick start button 601, the mobile phone displays buttons for multiple setting items of the input method on the interface of the note in response to the user's click operation, such as Figure 8 As shown in (d) in the figure. When the user clicks the more settings button 602, the mobile phone can also display the following information in response to the user's click operation: Figure 8 The setting interface of XX input method shown in (e).

[0114] The above content uses a mobile phone as an example to introduce the entry point of the breath-activated voice-to-text function. After the breath-activated voice-to-text function is enabled, in application scenarios where the mobile phone opens the input method and needs to enter text, the mobile phone can execute the input interaction method provided in the embodiments of this application to collect the user's audio data and convert it into text for display.

[0115] Of course, the input interaction method provided in the embodiments of the present application can also be applied to electronic devices such as tablet computers, personal digital assistants (PDAs), desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0116] Take mobile phones as an example. Figure 9 This is an example of the composition of an electronic device provided in an embodiment of the present application. Figure 9As shown, the electronic device 100 may include a processor 110, an internal memory 120, a camera 130, a display screen 140, a mobile communication module 150, a wireless communication module 160, an audio module 170, and a sensor module 180, etc.

[0117] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0118] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a smart sensor hub, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0119] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0120] The internal memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0121] In some embodiments, the internal memory 120 stores instructions for executing the input interaction method. The processor 110 can execute the instructions stored in the internal memory 120 to detect that the electronic device meets the conditions for the breath-activated speech-to-text function of the input method, collect the user's audio data, and convert it into text for display.

[0122] The electronic device implements display functionality through a GPU, display screen 140, and an application processor. A GPU is a microprocessor for image processing that connects display screen 140 and the application processor. The GPU performs mathematical and geometric calculations for image rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0123] Display screen 140 is used to display images, video interfaces, and the like. Display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 140, where N is a positive integer greater than one.

[0124] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0125] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0126] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 110 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0127] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 150 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0128] The electronic device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0129] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0130] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The electronic device can listen to music or make hands-free calls through the speaker 170A.

[0131] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear.

[0132] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device can be provided with at least one microphone 170C. In other embodiments, the electronic device can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0133] In some embodiments, the at least one microphone 170C provided in the electronic device is located at the bottom of the electronic device and is used to collect voice signals input by the user.

[0134] The headphone jack 170D is used to connect a wired headphone and can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0135] In the sensor module 180, the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 140. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device including at least two parallel plates with conductive material. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 140, the electronic device detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0136] Touch sensor 180B, also known as a "touch-sensitive device," can be disposed on display screen 140. Touch sensor 180B and display screen 140 form a touch screen, also known as a "touch screen." Touch sensor 180B is configured to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 140. In other embodiments, touch sensor 180B can also be disposed on the surface of the electronic device, at a location different from that of display screen 140.

[0137] The acceleration sensor 180C can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity and can also be used to identify the posture of the electronic device.

[0138] The gyro sensor 180D can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180D.

[0139] In addition, the electronic device runs an operating system on top of the above components. operating system, operating system, Operating system, etc. Applications can be installed and run on the operating system.

[0140] Figure 10 A schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0141] The layered architecture divides the operating system of an electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the operating system of the electronic device is the Android system. The Android system can be divided into five layers: from top to bottom: the application (APP) layer, the application framework layer (FWK), the system library, the hardware abstraction layer (HAL), and the kernel layer.

[0142] The application layer can include a series of application packages. Figure 10 As shown, the application package may include applications such as instant messaging applications, notes, breath wake-up engines, input methods, voice assistants, and perception modules.

[0143] The application framework layer provides an application programming interface (API) and programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0144] like Figure 10 As shown, the application framework layer may include window management services, content providers, phone managers, view systems, resource managers, speech recognition frameworks (HwSoundTrigger), multimodal interaction frameworks, and input managers (InputManager), etc.

[0145] The window management service is used to manage window programs. The window management service can implement window addition, deletion, display and hiding controls, etc. The content provider is used to store and obtain data and make this data accessible to applications. The data can include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions for electronic devices. For example, the management of call status (including answering, hanging up, etc.). The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0146] The speech recognition framework can provide the lower-level modules with instructions sent by the breath wake-up engine, and can also provide the breath wake-up engine with events reported by the lower level.

[0147] The multimodal interaction framework uses the Eagle Eye module to monitor the user's gaze and, if it determines that the user's gaze is in the area of ​​a banner notification, opens the corresponding app interface. If the corresponding app interface is currently open, the multimodal interaction framework can launch the input method and display the input method keyboard in the corresponding app interface.

[0148] The input manager can monitor the input method launch event generated when the input method is launched.

[0149] The Android Runtime includes core libraries and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system. In some embodiments of the present application, an application cold start will be executed within the Android Runtime. The Android Runtime can then obtain the application's optimization file status parameters. The Android Runtime can then use the optimization file status parameters to determine whether the optimization file has become outdated due to a system upgrade and return the determination result to the application management module.

[0150] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0151] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0152] The system library may include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0153] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0154] The HAL sits between the kernel layer and the application framework layer, serving as a bridge between them. In some embodiments, the HAL layer includes the speech recognition HAL (sound trigger HAL) and the Hawkeye module. The speech recognition HAL is used to enable or disable the breath wake-up algorithm module and report breath wake-up events to the speech recognition framework. The Hawkeye module is used to monitor the user's gaze.

[0155] The kernel layer is the layer between hardware and software. It includes at least a display driver, a sensor driver, an audio driver, and a breath-activated wake-up algorithm module. In some embodiments, the display driver controls the display screen's image display. The sensor driver controls the operation of multiple sensors, such as accelerometers, gyroscopes, pressure sensors, and touch sensors. The audio driver caches user input audio data, and the breath-activated wake-up algorithm module monitors whether the user raises their hand while holding the phone and inputs audio data.

[0156] Under the above five-layer architecture, the electronic device is also provided with a hardware layer, which may include the aforementioned electronic device hardware components, such as acceleration sensors, gyroscope sensors, pressure sensors and touch sensors. For example, Figure 10 Showing the display and camera.

[0157] It should be noted that although the embodiments of the present application are based on The system is used as an example, but the basic principles are also applicable to Electronic devices with other operating systems.

[0158] The technical solutions involved in the following embodiments can all be implemented in electronic devices having the above-mentioned hardware architecture and software architecture.

[0159] As explained in the aforementioned breath wake-up terminology, the breath wake-up function can be provided to voice assistants or input methods. That is, electronic devices can use breath wake-up to implement the breath wake-up voice assistant function, and can also use breath wake-up to implement breath wake-up speech-to-text. However, the breath wake-up function cannot be provided to both voice assistants and input methods at the same time. Therefore, electronic devices need to determine the specific voice input service that the breath wake-up function evokes based on specific business scenario requirements, such as evoked the voice assistant or input method's speech-to-text function based on the business scenario.

[0160] In some embodiments, the electronic device may be configured with multiple switch buttons for the breath wake-up function, with the voice assistant and input method corresponding to one switch button respectively. Of course, other switch buttons may also correspond to other applications of the electronic device, providing the breath wake-up function for other applications. Among the multiple switch buttons for the breath wake-up function, only one switch button may be in the selected state at a time. For example, the unselected switch button is grayed out and cannot be operated, and the switch button in the selected state corresponds to the corresponding application that can use the breath wake-up function. In this way, the user can independently select the application provided by the breath wake-up function by selecting a switch button, which is simple and convenient to operate.

[0161] In other embodiments, the electronic device is configured with switches for the voice assistant and input method respectively to use the breath wake-up function. The voice assistant is configured with a switch for the breath wake-up voice assistant function, and the input method is configured with a switch for the breath wake-up speech-to-text function. Users can independently select the application provided by the breath wake-up function by operating the switch. Figures 4 to 7 The following shows how to configure the switch of the breath-awakened voice-to-text function in the input method. The voice assistant can also configure the switch of the breath-awakened voice assistant function. Figure 6 The present embodiment does not provide detailed descriptions of this method.

[0162] The user turns on the breath-awakening voice-to-text function by pressing the button, and the breath-awakening voice-to-text function of the input method is activated. When the electronic device determines that the input method is pulled up to run, the electronic device synchronously executes the breath-awakening function and provides the input method with the running result of the breath-awakening function. If the user's lifting action and voice input are detected, the input method converts the user's input voice into text and displays it in the text input box of the input method. The user turns off the breath-awakening voice-to-text function by pressing the button, and the breath-awakening voice-to-text function of the input method is turned off. The electronic device will not execute the breath-awakening voice-to-text function, and the input method will not convert the voice input by the user. For example, Figure 3A and Figure 3B The button for users to turn on the breath-awakened speech-to-text function is displayed. During operation, the input method uses the results of the breath-awakening function to convert the user's input speech into text and display it in the text input box of the input method.

[0163] Similarly, the user turns on the switch button of the breath awakening voice assistant function to start the breath awakening voice assistant function of the voice assistant. The electronic device executes the breath awakening function and provides the voice assistant with the running result of the breath awakening function, and the voice assistant responds to the voice interaction instruction input by the user. The user turns off the switch button of the breath awakening voice assistant function to turn off the breath awakening voice assistant function of the voice assistant. The electronic device will not execute the breath awakening voice assistant function, and the voice assistant will not respond to the voice interaction instruction input by the user. For example, Figure 3D The button for users to turn on the breath-awakening voice assistant function is shown. The voice assistant uses the results of the breath-awakening function to respond to the user's voice interaction commands for querying the weather and display the weather information on the display screen.

[0164] It should be noted that the switch for the breath-activated speech-to-text function configured in the input method and the switch for the breath-activated voice assistant function configured in the voice assistant are mutually exclusive switches. At any given moment, only one of the switches for the breath-activated speech-to-text function and the breath-activated voice assistant function is in the on state. In some embodiments, the switch for the breath-activated speech-to-text function configured in the input method is in the off state, while the switch for the breath-activated voice assistant function configured in the voice assistant can be turned on by the user; when the switch for the breath-activated speech-to-text function configured in the input method is turned on, the switch for the breath-activated voice assistant function configured in the voice assistant is simultaneously turned off.

[0165] In other embodiments, the electronic device configures a switch for the breath awakening function for the voice assistant and the input method respectively. The voice assistant configures a switch for the breath awakening voice assistant function, and the input method configures a switch for the breath awakening voice-to-text function. The input method can also configure a switch for the breath awakening voice-to-text function as follows: Figures 4 to 7 As shown, the voice assistant can also be configured to wake up the voice assistant function by breathing. Figure 6 way.

[0166] When the user turns on the breath-activated voice assistant function and the breath-activated speech-to-text function, the voice assistant and input method can register their services with the breath-activated engine. The breath-activated engine can activate the breath-activated function based on the registered services and provide the breath-activated function results to the voice assistant or input method based on their service needs. This simplifies user operations and intelligently implements the breath-activated function based on the service needs of the input method and voice assistant.

[0167] Figure 11 This diagram shows the principles of the input method and voice assistant registration services.

[0168] like Figure 11 As shown, the breath wake-up engine may include a switch state management module and a scenario perception decision module. The switch state management module is used to manage the application's registered services, and the scenario perception decision module is used to determine the current application scenario. When the user turns on the switch of the input method's breath wake-up speech-to-text function, the input method registers the service with the switch state management module of the breath wake-up engine in response to the user's operation. Similarly, when the user turns on the switch of the voice assistant's breath wake-up voice assistant function, the voice assistant registers the service with the switch state management module in response to the user's operation.

[0169] Among them: the input method or voice assistant service registers the service with the switch state management module of the breath wake-up engine, which can be understood as the input method or voice assistant service registering to use the breath wake-up function. The switch of the breath wake-up speech-to-text function of the input method is turned on by the user, including: the user Figures 4 to 7 Use any of the provided methods to enable the breath-activated speech-to-text function of the input method.

[0170] If the switch state management module determines that there is a registered service, the breath awakening path is opened. Of course, if the switch state management module determines that there is no registered service, the breath awakening path is closed.

[0171] The breath wake-up path is opened, and the breath wake-up algorithm module is enabled to run synchronously to detect the user's hand-raising action while holding the mobile phone and whether the user inputs audio data. In some embodiments, the breath wake-up algorithm module can detect the user's hand-raising action while holding the mobile phone through motion data collected by the acceleration sensor and / or the gyroscope sensor, and can collect the user's audio data through the microphone. When the breath wake-up algorithm module detects the user's hand-raising action and collects audio data, a breath wake-up event is generated, and the breath wake-up event is reported to the scene perception decision module through the speech recognition hal and the speech recognition framework. The breath wake-up event is used to indicate that the user is speaking close to the microphone.

[0172] The scene perception decision module performs scene query based on the registered business, that is, queries the status of the application corresponding to the registered business. Furthermore, the scene perception decision module distributes breath wake-up events to the application according to the business priority requirements of the application. The business priority of the application refers to the priority of the application using the breath wake-up function to execute the business. In some embodiments, the business priority of the input method is higher than the priority of the voice assistant business. When the scene perception decision module finds that the input method is pulled up and running, it distributes a breath wake-up event to the input method to wake up the speech-to-text function of the input method. Otherwise, it distributes a breath wake-up event to the voice assistant.

[0173] The following combination Figure 12 , for input method Figure 11 After the business registration is completed in the schematic diagram shown, the input method executes the input interaction method for introduction.

[0174] like Figure 12 As shown, the input interaction method includes:

[0175] S101. The input manager reports an input method launch event to the scene perception decision module of the breath wake-up engine.

[0176] The input method is pulled up and run, which can be understood as Figure 3A In (c), the keyboard is pulled up and the cursor is in the text input box. In some embodiments, Figure 10 As shown, the input manager of the application framework layer can sense that the input method is pulled up and run, generate an input method pull-up event, and the input manager reports the input method pull-up event to the scene perception decision module of the breath wake-up engine.

[0177] S102: The sensor collects motion data.

[0178] The sensor in this step may refer to an acceleration sensor and / or a gyroscope sensor.

[0179] In some embodiments, as mentioned above Figure 11Based on the content of the input method, the input method registers its service with the switch state management module. The switch state management module enables the breath wake-up path and enables the breath wake-up algorithm module. The breath wake-up algorithm module can detect the user's hand-raising gesture while holding the phone using motion data collected by the accelerometer and / or gyroscope sensor. Based on this, the accelerometer and / or gyroscope sensor are activated to collect motion data.

[0180] In other embodiments, the user activates the breath-activated speech-to-text function by turning on a switch for the breath-activated speech-to-text function. When the user turns on the breath-activated speech-to-text function switch, the breath-activated path is also activated by the breath-activated engine, and the breath-activated algorithm module is enabled to run. The breath-activated algorithm module can also detect the user's hand-raising motion while holding the phone using motion data collected by an accelerometer and / or gyroscope sensor.

[0181] S103: The audio driver caches the audio data.

[0182] As described in the aforementioned step S102, the breath awakening algorithm module is enabled to run. The breath awakening algorithm module can also collect the user's audio data through the microphone, and the audio driver of the electronic device caches the user's audio data.

[0183] The audio driver of the electronic device can cache the audio data of the user input collected within a preset time before the current time, and the cached audio data can be refreshed periodically, and the refresh period is shorter than the preset time. Exemplarily, the preset time can be 6 seconds, and the refresh period can be 1 second. For example, assuming that the current time is 00:15:13, the audio driver caches the audio data from 00:15:08 to 00:15:13. At 00:15:14, the audio driver deletes the audio data from 00:15:08 to 00:15:09 and stores the audio data from 00:15:13 to 00:15:14, that is, the audio driver caches the audio data from 00:15:09 to 00:15:14. The audio data cached by the audio driver can be used to perform audio-to-text conversion.

[0184] The above method of caching audio data is merely an example of an embodiment of the present application, and the present application is not limited thereto.

[0185] It should be noted that there is no specific restriction on the execution order of step S102 and step S103, and they can also be understood as being performed simultaneously.

[0186] S104: The audio driver sends audio data to the breath awakening algorithm module.

[0187] The audio driver can send the cached audio data to the breath wake-up algorithm module.

[0188] S105. The sensor sends motion data to the breath algorithm wake-up module.

[0189] The sensor collects motion data and sends it to the breath calculation wake-up method module. Of course, the sensor in this step can also refer to an acceleration sensor and / or a gyroscope sensor.

[0190] It should be noted that there is no specific limitation on the execution order of step S104 and step S105.

[0191] S106: The breath awakening algorithm module generates a breath awakening event.

[0192] like Figure 3A (c) and (d), or Figure 3B As shown in (c) and (d), in an application scenario where the input method is launched, the user raises the phone and inputs audio data. The breath wake-up algorithm module detects the user raising the phone and inputting audio data through the audio data sent by the audio driver and the motion data sent by the sensor, and generates a breath wake-up event. In some embodiments, the breath wake-up event generated by the breath wake-up algorithm module may include a session identifier (SessionID), which is used to obtain the audio data.

[0193] S107. The breath awakening algorithm module reports the breath awakening event to the speech recognition HAL.

[0194] Because the breath awakening path is opened, after the breath awakening algorithm module generates a breath awakening event, it can report the breath awakening event to the speech recognition HAL.

[0195] S108. The speech recognition hal reports the breath awakening event to the speech recognition framework.

[0196] The speech recognition hal receives the breath awakening event reported by the breath awakening algorithm module and reports the breath awakening event to the speech recognition framework.

[0197] S109. The speech recognition framework reports the breath awakening event to the scene perception decision module.

[0198] The speech recognition framework receives the breath awakening event reported by the speech recognition HAL and reports the breath awakening event to the scene perception decision module.

[0199] S110. The scene perception decision module detects that the input method is pulled up.

[0200] The context-aware decision module queries the status of the application corresponding to the registered service. As shown in step S101, the context-aware decision module receives an input method launch event reported by the input manager. The context-aware decision module detects that the input method has been launched and needs to dispatch a breath wake-up event to the input method to activate the input method's speech-to-text function.

[0201] In other scenarios, such as Figure 3E In the scenario shown, the scene perception decision module detects that the input method has not been pulled up, such as not receiving the input method pull-up event reported by the input manager. The scene perception decision module sends a breath wake-up event to the voice assistant, and the voice assistant is enabled to run to obtain the audio data input by the user and respond to the voice interaction instructions indicated by the audio data.

[0202] It should be noted that Figure 12 The following shows an execution order of step S109 and step S110. It is known that step S109 and step S110 are not limited to Figure 12 The execution order shown may be executed in parallel, or step S110 may be executed first and then step S109.

[0203] S111. The scene perception decision module sends a breath wake-up event to the input method.

[0204] When the scene perception decision module detects that the input method is pulled up and running, it sends a breath wake-up event to the input method.

[0205] In some embodiments, the user activates the breath-wake-up speech-to-text function by turning on the switch of the breath-wake-up speech-to-text function. Steps S109 to S111 may not be executed. The breath-wake-up engine receives the breath-wake-up event reported by the speech recognition framework and sends the breath-wake-up event to the input method.

[0206] S112: The input method starts a speech-to-text function and obtains cached audio data through an audio driver.

[0207] The input method needs to convert the audio data input by the user into text. Therefore, the input method needs to obtain the cached audio data from the audio driver. In some embodiments, the input method receives a breath wake-up event, obtains the session ID carried in the breath wake-up event, specifies the session ID to match the audio driver's path, and obtains the audio data cached by the audio driver.

[0208] S113: The input method converts the audio data into corresponding text.

[0209] In some embodiments, the input method may utilize a speech processing model to convert the acquired audio data into text to obtain the corresponding text. This embodiment does not describe the specific implementation process of converting audio data into text.

[0210] S114: The input method controls the display of text.

[0211] After the input method obtains the text corresponding to the audio data, it can control the display of the text. In some embodiments, the input method controls the text to be displayed in the text input box. Figure 3A As shown in (e).

[0212] In this embodiment, the breath wake-up algorithm module detects the user raising their hand while holding the phone and collects user input audio data, generating a breath wake-up event. The scene perception decision module receives the breath wake-up event and detects that the input method has been launched, predicting that the user needs to enter text. The scene perception module then sends the breath wake-up event to the input method, triggering it to obtain the user's audio input data and convert it into text for display. This allows the user to enter text into the phone simply by voice input, without having to manually operate buttons on the display screen. This is simple and convenient operation.

[0213] Instant messaging applications will call input methods during operation, so instant messaging applications can be understood as associated applications of input methods. Figure 3B and Figure 3C From the displayed content, we can see that: if an application associated with an input method, such as an instant messaging application, generates a banner notification, and the interface of the application corresponding to the banner notification can display the input method's keyboard, the phone will use the eye tracking function to detect that the gaze point is in the area where the banner notification is located. The phone will automatically open the application corresponding to the banner notification and start the input method. In addition, the breath-awakened speech-to-text function of the input method is started and running. The phone uses the breath-awakened speech-to-text function to collect the user's input voice and convert the voice into corresponding text.

[0214] The following combination Figure 13 , the functional modules of the input method and the eye tracking function cooperate to complete the input interaction method. Of course, in some embodiments, the input method performs Figure 13 Before the solution is presented, you can also use Figure 11 The schematic diagram displayed completes the business registration.

[0215] like Figure 13 As shown, the input interaction method includes:

[0216] S201. The multimodal interaction framework registers the scene fence with the perception module.

[0217] The multimodal interaction framework can be understood as a functional module of the eye tracking function, located in the application framework layer. After the eye tracking function is enabled, the multimodal interaction framework registers the scene fence with the perception module. In some embodiments, registering the scene fence can be understood as registering to monitor whether to generate a banner notification.

[0218] The perception module can be understood as an application module in the application layer, which is used to monitor the status of electronic devices such as mobile phones, such as monitoring whether the electronic devices generate banner notifications.

[0219] S202: The perception module sends a notification message to the multimodal interaction framework to notify that the scene fence condition is met.

[0220] After the multimodal interaction framework registers the scene fence with the perception module, the perception module can monitor whether the electronic device meets the scene fence conditions. As mentioned in the previous steps, registering the scene fence is to register and monitor whether a banner notification is generated. Therefore, the scene fence condition can be understood as the electronic device generating a banner notification and displaying the banner notification.

[0221] When the perception module detects that the electronic device generates and displays a banner notification, it sends a notification message to the multimodal interaction framework to notify the multimodal interaction framework that the electronic device meets the scene fence condition.

[0222] S203: The multi-mode interaction framework controls the operation of the Hawkeye module.

[0223] When the multimodal interaction framework receives the notification message, it sends a control instruction to the Hawkeye module to control the operation of the Hawkeye module.

[0224] The Eagle Eye module can also be understood as a functional module of the eye tracking function, located in the hardware abstraction layer.

[0225] S204: The Eagle Eye module detects eye movement and gazes at a banner notification.

[0226] The Eagle Eye module runs to detect whether the user's gaze is within a specific area of ​​the display. This specific area can be understood as the area where the banner notification is displayed. If the Eagle Eye module detects that the user's gaze is within the specific area of ​​the display, it indicates that the user is looking at the banner notification; otherwise, the user is not looking at the banner notification.

[0227] If the Eagle Eye module detects an eye gaze banner notification, it can control the multimodal interaction framework to execute step S205; otherwise, it can exit this process. In some embodiments, if the Eagle Eye module detects an eye gaze banner notification, it can send a notification message to the multimodal interaction framework to inform the multimodal interaction framework of the detection result.

[0228] S205: The multi-modal interaction framework launches the input method.

[0229] The multi-modal interaction framework pulls up the input method, which can be understood as the multi-modal interaction framework pulling up the input method to run, that is, Figure 3A In (c), the keyboard is pulled up and the cursor is in the text input box.

[0230] Furthermore, the multimodal interaction framework will also pull up the banner notification to run the corresponding application. Of course, the interface of the application corresponding to the banner notification will also be displayed on the display.

[0231] In some embodiments, before executing this step, the multimodal interaction framework can determine that the user has raised their hand while holding the electronic device and inputs audio data. In this way, the multimodal interaction framework can know that the user intends to input voice and can launch the input method. The interface of the input method is displayed in the interface of the application corresponding to the banner notification. For example, Figure 3B As shown in (c).

[0232] S206: The input manager reports the input method launch event to the scene perception decision module of the breath wake-up engine.

[0233] As in the aforementioned step S101, the input method is launched and run, and the input manager of the application framework layer can sense that the input method is launched and run, and generate an input method launch event. The input manager reports the input method launch event to the scene perception decision module of the breath awakening engine to inform the scene perception decision module that the input method is in running state.

[0234] S207: The sensor collects motion data.

[0235] The content of step S207 can refer to the content of the aforementioned step S102 and will not be repeated here.

[0236] S208: The audio driver caches the audio data.

[0237] The content of step S208 can refer to the content of the aforementioned step S103 and will not be repeated here.

[0238] There is no specific limitation on the execution order of step S207 and step S208, which can also be understood as being performed simultaneously.

[0239] S209: The audio driver sends audio data to the breath awakening algorithm module.

[0240] The content of step S209 can refer to the content of the aforementioned step S104 and will not be repeated here.

[0241] S210: The sensor sends motion data to the breath algorithm wake-up module.

[0242] The content of step S210 can refer to the content of the aforementioned step S105 and will not be repeated here.

[0243] It should be noted that there is no specific limitation on the execution order of step S209 and step S210.

[0244] S211. The breath awakening algorithm module generates a breath awakening event.

[0245] The content of step S211 can refer to the content of the aforementioned step S106 and will not be repeated here.

[0246] S212. The breath awakening algorithm module reports the breath awakening event to the scene perception decision module through the speech recognition hal and speech recognition framework.

[0247] In some embodiments, the breath awakening algorithm reports the breath awakening event to the speech recognition HAL, the speech recognition HAL receives the breath awakening event, reports the breath awakening event to the speech recognition framework, and the speech recognition framework receives the breath awakening event and reports the breath awakening event to the scene perception decision module.

[0248] S212: The scene perception decision module detects that the input method is launched.

[0249] The content of step S212 can refer to the content of the aforementioned step S110 and will not be repeated here.

[0250] S213. The scene perception decision module sends a breath awakening event to the input method.

[0251] The content of step S213 can refer to the content of the aforementioned step S111 and will not be repeated here.

[0252] S214: The input method starts the speech-to-text function and obtains the cached audio data through the audio driver.

[0253] The content of step S214 can refer to the content of the aforementioned step S112 and will not be repeated here.

[0254] S215: The input method converts the audio data into corresponding text.

[0255] The content of step S215 can refer to the content of the aforementioned step S113 and will not be repeated here.

[0256] S216: The input method controls the display of text.

[0257] The content of step S216 can refer to the content of the aforementioned step S114 and will not be repeated here.

[0258] In this embodiment, the perception module detects that the electronic device generates and displays a banner notification, and then controls the Eagle Eye module through the multi-modal interaction framework to detect the user's gaze point. When it is detected that the user's gaze point is in the area of ​​the banner notification, it means that the user is looking at the banner notification displayed on the display screen, and there is a need to open the application of the banner notification, and there may also be obstacles to manual interaction with the banner notification. Furthermore, the user holds the electronic device and raises his hand, and inputs voice data, and the multi-modal interaction framework starts the input method to run, simplifying the user operation. Further, the scene perception module sends a breath wake-up event to the input method, triggering the input method to obtain the audio data input by the user, and converting it into text display, so that the user does not need to manually operate the buttons on the display screen, and can input text into the mobile phone only by inputting voice, which is simple and convenient to operate.

[0259] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0260] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, a non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0261] Another embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

Claims

1. An input interaction method, characterized in that: include: The electronic device displays a first interface, wherein the first interface includes a first text input area; During the process of the electronic device displaying the first interface, when the user lifts the electronic device to the mouth and speaks, the first speech-to-text function is turned on, and the electronic device displays the text corresponding to the user input speech in the first text input area; In response to the user's operation on the first text input area, the electronic device displays a second interface, the second interface comprising: a speech-to-text button and a second text input area; When the electronic device displays the second interface, the second voice-to-text function is turned on when the user presses the voice-to-text button, and when the user inputs voice, the text corresponding to the user input voice is displayed in the second text input area.

2. The input interaction method according to claim 1, characterized in that: When the user lifts the electronic device to his mouth to speak, the user does not operate the electronic device.

3. The input interaction method according to claim 1, characterized in that: The first interface does not include a speech-to-text button, or the first interface does not include an interface of an input method application.

4. The input interaction method according to any one of claims 1 to 3, characterized in that: The first interface also includes an interface of a first application.

5. The input interaction method according to claim 4, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a third interface, where the third interface includes a first banner notification of the first application; The electronic device displays the first interface, including: displaying the first interface in response to a click operation of the user's finger on the first banner notification.

6. The input interaction method according to claim 4, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a third interface, where the third interface includes a first banner notification of the first application; The electronic device displays a first interface, comprising: in a process in which the electronic device displays the third interface, in response to a user gazing at the first banner notification, the electronic device displays the first interface.

7. The input interaction method according to any one of claims 1 to 6, characterized in that: In response to the user's operation on the text input area of ​​the first interface, the electronic device displays the second interface, including: In response to a user clicking operation on the first text input area, the electronic device displays the second interface.

8. The input interaction method according to claim 1, characterized in that: Also includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth and speaks, the electronic device invokes a voice assistant, wherein the fourth interface does not include a text input area.

9. The input interaction method according to any one of claims 1 to 8, characterized in that: The electronic device is provided with a first switch, and the first switch is in an on state.

10. The input interaction method according to any one of claims 1 to 8, characterized in that: The electronic device is provided with a first switch, the first switch is in a closed state, and the method further includes: When the electronic device displays the first interface, when the user lifts the electronic device to his mouth and speaks, the electronic device does not activate the first speech-to-text function.

11. The input interaction method according to claim 8, characterized in that: The electronic device is provided with a second switch, and the second switch is in an on state.

12. The input interaction method according to claim 8, characterized in that: The electronic device is provided with a second switch, the second switch is in a closed state, and the method further includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth to speak, the electronic device does not wake up the voice assistant.

13. An input interaction method, characterized in that: include: The electronic device displays a first interface, wherein the first interface includes a first text input area and a speech-to-text button; During the process of the electronic device displaying the first interface, when the user lifts the electronic device to his mouth to speak, the electronic device turns on the first speech-to-text function, switches the speech-to-text button from a first state to a second state, and displays text corresponding to the user's speech input in the first text input area, wherein the user does not operate the speech-to-text button when the user lifts the electronic device to his mouth to speak.

14. The input interaction method according to claim 13, characterized in that: The first state is a closed state, the second state is a first open state, In the first on state, the electronic device continuously receives voice input from the user, and converts the voice input into text and displays it in the first text input area.

15. The input interaction method according to claim 13, characterized in that: The first state is the second on state, the second state is the first on state, In the second on state, when the electronic device detects that the user clicks the speech-to-text button, it receives the user's speech input and converts the speech input into text and displays it in the first text input area; In the first on state, the electronic device continuously receives voice input from the user, and converts the voice input into text and displays it in the first text input area.

16. The input interaction method according to claim 13, characterized in that: The first text input area of ​​the first interface also displays text.

17. The input interaction method according to claim 13, characterized in that: The first interface also includes an interface of a first application.

18. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In response to a user clicking operation on the first banner notification, displaying the third interface, wherein the third interface includes an interface of the first application and a second text input area; The electronic device displays the first interface including: displaying the first interface in response to a user clicking operation on the second text input area.

19. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In response to a user clicking operation on the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

20. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; The electronic device displays the first interface including: in the process of the electronic device displaying the second interface, in response to a user gazing at the first banner notification, the electronic device displays the first interface.

21. The input interaction method according to claim 20, characterized in that: When the user looks at the first banner notification, the user does not operate the screen of the electronic device.

22. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: displaying the first interface in response to a user clicking operation on the second text input area.

23. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

24. The input interaction method according to claim 23, characterized in that: When the user looks at the first banner notification, the user does not operate the screen of the electronic device.

25. The input interaction method according to claim 13, characterized in that: Also includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth to speak, the electronic device invokes a voice assistant, wherein the fourth interface does not include a text input area.

26. The input interaction method according to any one of claims 13 to 25, characterized in that: The electronic device is provided with a first switch, and the first switch is in an on state.

27. The input interaction method according to any one of claims 13 to 25, characterized in that: The electronic device is provided with a first switch, the first switch is in a closed state, and the method further includes: When the electronic device displays the first interface, when the user lifts the electronic device to his mouth and speaks, the electronic device does not activate the first speech-to-text function.

28. The input interaction method according to claim 25, characterized in that: The electronic device is provided with a second switch, and the second switch is in an on state.

29. An electronic device, characterized in that: include: one or more processors, memory, and display screens; The memory and the display screen are coupled to the one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the input interaction method as described in any one of claims 1 to 12, or the input interaction method as described in any one of claims 13 to 28.

30. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the input interaction method as described in any one of claims 1 to 12, or the input interaction method as described in any one of claims 13 to 28.

Images