Input interaction method, electronic device and readable medium

Abstract

Provided in the present application are an input interaction method, an electronic device and a readable medium. During the process of the electronic device displaying a first interface, when a user lifts the electronic device to be in close proximity to the mouth and speaks, a first speech-to-text function is enabled, and the electronic device displays text corresponding to speech, which is inputted by the user, in a first text input area. The present application is applied to application scenarios where the user views a notification, inputs text, etc. An interface, which is displayed by the electronic device, comprises a first text input area. A user may lift the electronic device to be in close proximity to the mouth and speak, and the electronic device enables a first speech-to-text function, such that the electronic device can display text corresponding to speech, which is inputted by the user, in the first text input area, without needing the user to additionally operate a screen of the electronic device, thereby simplifying an operation of the user on the screen, and being convenient for the user.

Description

Input interaction method, electronic device and readable medium

[0001] This application claims priority to the Chinese patent application filed with the State Intellectual Property Office of China on June 15, 2023, with application number 202310713948.0 and invention name “Input interaction method, electronic device and readable medium”, the entire contents of which are incorporated by reference into this application. Technical Field

[0002] The present application relates to the field of data processing technology, and in particular to an input interaction method, electronic device, computer program product, and computer-readable storage medium. Background Art

[0003] When users hold electronic devices such as mobile phones in one hand, they usually use their thumbs to input and click on the display screen. However, the thumb's operating range is difficult to cover both the top and bottom of the device at the same time.

[0004] In application scenarios such as when users view notifications and input text, the user's operations involve the top and bottom of the device. If the user uses the electronic device by holding the device with one hand, the thumb usually cannot reach both the top and bottom of the device, which brings great inconvenience to the user's operation.

[0005] Summary of the Invention

[0006] The present application provides an input interaction method, electronic device, computer program product and computer-readable storage medium, the purpose of which is to simplify user operations on the screen in application scenarios such as notifications and text input, thereby bringing convenience to users.

[0007] In order to achieve the above objectives, this application provides the following technical solutions:

[0008] In a first aspect, the present application provides an input interaction method, including: an electronic device displays a first interface, the first interface includes a first text input area; during the process of the electronic device displaying the first interface, when the user lifts the electronic device to his mouth to speak, a first voice-to-text function is turned on, and the electronic device displays the text corresponding to the user input voice in the first text input area; in response to the user's operation on the first text input area, the electronic device displays a second interface, the second interface includes: a voice-to-text button and a second text input area; during the process of the electronic device displaying the second interface, when the user presses the voice-to-text button, the second voice-to-text function is turned on, and when the user input voice is detected, the text corresponding to the user input voice is displayed in the second text input area.

[0009] It can be seen from the above content that: in application scenarios such as users viewing notifications and entering text, the interface displayed by the electronic device includes a first text input area. The user can lift the electronic device and put it to his mouth to speak. The electronic device turns on the first voice-to-text function. The user does not need to operate the screen of the electronic device additionally. The electronic device can display the text corresponding to the voice input by the user in the first text input area, which simplifies the user's operation of the screen and brings convenience to the user.

[0010] In some embodiments, the first speech-to-text function is a breath-activated speech-to-text function, and the second speech-to-text function is a speech-to-text function activated by the user pressing a speech-to-text button. The first speech-to-text function has stronger noise immunity than the second speech-to-text function, and can largely block out interference signals during the user's speech input.

[0011] In one possible implementation, when the user lifts the electronic device and puts it to his mouth to speak, the user does not operate the electronic device.

[0012] In a possible implementation, the first interface does not include a speech-to-text button, or the first interface does not include an interface of an input method application.

[0013] In a possible implementation, the first interface also includes an interface of a first application.

[0014] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays a third interface, the third interface includes a first banner notification of the first application; wherein the electronic device displays the first interface, including: displaying the first interface in response to a click operation of the user's finger on the first banner notification.

[0015] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays a third interface, the third interface includes a first banner notification of the first application; the electronic device displays the first interface, including: in the process of the electronic device displaying the third interface, in response to the user looking at the first banner notification, the electronic device displays the first interface.

[0016] In this possible implementation, the electronic device displays a third interface, which includes a first banner notification of the first application. The user does not need to operate the screen of the electronic device but only looks at the first banner notification. The electronic device can then display the first interface, further facilitating user operation.

[0017] In one possible implementation, in response to a user operation on a text input area of ​​a first interface, the electronic device displays a second interface, including: in response to a user click operation on the first text input area, the electronic device displays the second interface.

[0018] In one possible implementation, it also includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to his mouth to speak, the electronic device wakes up the voice assistant, wherein the fourth interface does not include a text input area. In this possible implementation, the fourth interface may be an interface of the electronic device that is different from the first interface. In some embodiments, the fourth interface does not include an interface for an input method application. When the electronic device displays the fourth interface or the electronic device is in the screen-off state, the user lifts the electronic device to his mouth to speak to wake up the voice assistant, and the voice assistant responds to the instructions corresponding to the user's input voice. In some embodiments, when the user lifts the electronic device to his mouth to speak, the electronic device waking up the voice assistant can be called a breath-awakening voice assistant function.

[0019] In one possible embodiment, the electronic device is provided with a first switch, and the first switch is in an on state. In this possible embodiment, the first switch corresponds to a first speech-to-text function. When the switch is in the on state, the first speech-to-text function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0020] In one possible implementation, the first switch is set on the interface of the first setting item of the input method application; wherein, when the first switch is turned on, the interface of the first setting item displays a first pop-up window, and the first pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0021] In a possible implementation, the electronic device is configured with a first switch setting item on the interface of the input method application, and the interface of the first switch setting item includes a button of the first switch and a guide animation.

[0022] In one possible embodiment, the first switch is a speech-to-text button of an input method application. The duration of the speech-to-text button being pressed for the first time is greater than a threshold. The electronic device displays a second pop-up window. The second pop-up window includes a first text control. The first text control is clicked, the function corresponding to the first switch is turned on, and a third pop-up window is displayed. The third pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0023] In one possible implementation, the electronic device is provided with a first switch, and the first switch is in an off state. The method further includes: while the electronic device is displaying the first interface, when the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function. In this possible implementation, when the first switch is in the off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function, and the first text input area of ​​the electronic device does not display text, thereby disabling the first speech-to-text function.

[0024] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an on state. In this possible embodiment, the second switch is a switch for a breath-activated voice assistant function. When the switch is in the on state, the breath-activated voice assistant function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0025] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an off state. The method further includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to their mouth to speak, the electronic device does not awaken the voice assistant. In this possible embodiment, when the second switch is in an off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the breath-activated voice assistant function, and the voice assistant does not respond to commands corresponding to the user's voice input, thereby disabling the breath-activated voice assistant function.

[0026] In a second aspect, the present application provides an input interaction method, including: an electronic device displays a first interface, the first interface including a first text input area and a speech-to-text button; during the process of the electronic device displaying the first interface, when the user lifts the electronic device and puts it to his mouth to speak, the electronic device turns on the first speech-to-text function, switches the speech-to-text button from the first state to the second state, and displays the text corresponding to the user's voice input in the first text input area, wherein the user does not operate the speech-to-text button when the user lifts the electronic device and puts it to his mouth to speak.

[0027] It can be seen from the above content that: in application scenarios such as users viewing notifications and entering text, the interface displayed by the electronic device includes a first text input area and a voice-to-text button. The user can lift the electronic device and put it to his mouth to speak. The electronic device turns on the first voice-to-text function. The user does not need to operate the screen of the electronic device additionally. The electronic device can display the text corresponding to the voice input by the user in the first text input area, which simplifies the user's operation of the screen and brings convenience to the user.

[0028] In one possible embodiment, the first state is an off state, and the second state is a first on state. The method further includes: in the first on state, the electronic device continuously receives the user's voice input and converts the voice input into text and displays it in the first text input area. In this possible embodiment, the voice-to-text button is switched to the first on state, and the user lifts the electronic device to their mouth and continues to input voice. The first voice-to-text function is continuously activated, and the electronic device continuously receives the user's voice input and converts the voice input into text and displays it in the text input area. In this way, the electronic device converts the user's continuously input voice into text for display.

[0029] In one possible embodiment, the first state is the second on state, and the second state is the first on state. In the second on state, when the electronic device detects that the user clicks the voice-to-text button, it receives the user's voice input and converts the voice input into text and displays it in the first text input area; in the first on state, the electronic device continues to receive the user's voice input and converts the voice input into text and displays it in the first text input area.

[0030] In this possible implementation, the speech-to-text button includes a first on state and a second on state. In the first on state, when the user clicks the speech-to-text button, the electronic device can convert the speech input by the user into text and display it in the input text area. In the second on state, it indicates that the first speech-to-text function is activated. When the user lifts the electronic device and continues to input speech, the electronic device can convert the speech input by the user into text and display it in the first text input area.

[0031] In one possible implementation, the first text input area of ​​the first interface also displays text. In some embodiments, when the electronic device displays the first interface, the user lifts the electronic device to their mouth and speaks, and the first voice-to-text function is activated, and the electronic device displays the text corresponding to the user's input voice in the first text input area, or the user presses the voice-to-text button and activates the second voice-to-text function, and the electronic device displays the text corresponding to the user's input voice in the first text input area. In this way, the first text input area of ​​the first interface displayed by the electronic device also displays text. Moreover, when the user lifts the electronic device to their mouth and speaks again, the first voice-to-text function is activated, and the electronic device can continue to display the text corresponding to the user's input voice in the first text input area.

[0032] In a possible implementation, the first interface also includes an interface of a first application.

[0033] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes the first banner notification of the first application; in response to the user's click operation on the first banner notification, the third interface is displayed, the third interface includes the interface of the first application and the second text input area; the electronic device displays the first interface including: in response to the user's click operation on the second text input area, the first interface is displayed.

[0034] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; in response to the user's click operation on the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

[0035] In this possible implementation, the electronic device displays a first banner notification of the first application, which is usually located at the top of the electronic device. When the user clicks the first banner notification, the electronic device displays the third interface and then automatically displays the first interface. In this way, the user only needs to operate the top of the electronic device and does not need to operate the bottom of the electronic device, which brings great convenience to the user holding the electronic device with one hand.

[0036] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; the electronic device displays the first interface including: in the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the electronic device displays the first interface.

[0037] In this possible implementation, the electronic device displays a first banner notification of the first application. When the user looks at the first banner notification, the electronic device displays the first interface. The user does not need to operate the electronic device screen, thereby simplifying the user's operation of the screen.

[0038] In one possible implementation, when the user looks at the first banner notification, the user does not operate the screen of the electronic device.

[0039] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; in the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: in response to the user clicking the second text input area, the first interface is displayed.

[0040] In this possible implementation, the electronic device displays a first banner notification of the first application, which is usually located at the top of the electronic device. The user does not need to operate the top of the electronic device, but only needs to look at the first banner notification. The electronic device displays a third interface. The second text input area in the third interface is usually located at the bottom of the electronic device. The user clicks on the second text input area, and the electronic device displays the first interface. In this way, the user only needs to operate the bottom of the electronic device, and does not need to operate the top of the electronic device, which also brings great convenience to the scenario where the user holds the electronic device with one hand.

[0041] In one possible embodiment, before the electronic device displays the first interface, it also includes: the electronic device displays the second interface, the second interface includes a first banner notification of the first application; during the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, the third interface is displayed, and the third interface includes a second text input area; the electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

[0042] In this possible implementation, the electronic device displays the first banner notification of the first application. The user does not need to operate the top of the electronic device, but only needs to look at the first banner notification. The electronic device displays the third interface. After the electronic device displays the third interface, it automatically displays the first interface. In this way, the user does not need to operate the electronic device screen, and the operation is more convenient.

[0043] In one possible implementation, when the user looks at the first banner notification, the user does not operate the screen of the electronic device.

[0044] In one possible implementation, it also includes: when the electronic device displays the fourth interface or the screen is off, when the user lifts the electronic device to his mouth to speak, the electronic device wakes up the voice assistant, wherein the fourth interface does not include a text input area. In this possible implementation, the fourth interface may be an interface of the electronic device that is different from the first interface. In some embodiments, the fourth interface does not include an interface for an input method application. When the electronic device displays the fourth interface or the electronic device is in the screen-off state, the user lifts the electronic device to his mouth to speak to wake up the voice assistant, and the voice assistant responds to the instructions corresponding to the user's input voice. In some embodiments, when the user lifts the electronic device to his mouth to speak, the electronic device waking up the voice assistant can be called a breath-awakening voice assistant function.

[0045] In one possible embodiment, the electronic device is provided with a first switch, and the first switch is in an on state. In this possible embodiment, the first switch corresponds to a first speech-to-text function. When the switch is in the on state, the first speech-to-text function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0046] In one possible implementation, the first switch is set on the interface of the first setting item of the input method application; wherein, when the first switch is turned on, the interface of the first setting item displays a first pop-up window, and the first pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0047] In a possible implementation, the electronic device is configured with a first switch setting item on the interface of the input method application, and the interface of the first switch setting item includes a button of the first switch and a guide animation.

[0048] In one possible embodiment, the first switch is a speech-to-text button of an input method application. The duration of the speech-to-text button being pressed for the first time is greater than a threshold. The electronic device displays a second pop-up window. The second pop-up window includes a first text control. The first text control is clicked, the function corresponding to the first switch is turned on, and a third pop-up window is displayed. The third pop-up window is used to prompt that the function corresponding to the first switch is turned on.

[0049] In one possible implementation, the electronic device is provided with a first switch, and the first switch is in an off state. The method further includes: while the electronic device is displaying the first interface, when the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function. In this possible implementation, when the first switch is in the off state, the user lifts the electronic device to their mouth to speak, the electronic device does not activate the first speech-to-text function, and the first text input area of ​​the electronic device does not display text, thereby disabling the first speech-to-text function.

[0050] In one possible embodiment, the electronic device is provided with a second switch, and the second switch is in an on state. In this possible embodiment, the second switch is a switch for a breath-activated voice assistant function. When the switch is in the on state, the breath-activated voice assistant function is activated when the user lifts the electronic device to their mouth to speak, and is not activated otherwise.

[0051] In a third aspect, the present application provides an electronic device comprising: one or more processors, a memory, and a display screen; the memory and the display screen are coupled to the one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the input interaction method provided in the first aspect and any one of its possible implementations, or the input interaction method provided in the second aspect and any one of its possible implementations.

[0052] In a fourth aspect, the present application provides a computer-readable storage medium for storing a computer program. When the computer program is executed, it is specifically used to implement the input interaction method provided in the first aspect and any one of its possible implementations, or the input interaction method provided in the second aspect and any one of its possible implementations.

[0053] In a fifth aspect, the present application provides a computer program product, which, when running on a computer, enables the computer to execute the input interaction method provided in the first aspect and any one of its possible implementations, or the input interaction method provided in the second aspect and any one of its possible implementations. BRIEF DESCRIPTION OF THE DRAWINGS

[0054] FIG1 is a diagram showing a user holding a mobile phone in one hand according to an embodiment of the present application;

[0055] FIG2 is an application scenario diagram of the input interaction method provided in an embodiment of the present application;

[0056] 3A to 3E illustrate several application scenarios of the breath awakening function provided in an embodiment of the present application;

[0057] Figures 4 to 7 are diagrams showing the activation of the breath-awakening speech-to-text function provided in an embodiment of the present application;

[0058] FIG8 is a diagram showing another way of entering the interface of the input method provided in an embodiment of the present application;

[0059] FIG9 is a hardware structure diagram of an electronic device provided in an embodiment of the present application;

[0060] FIG10 is a software framework diagram of an electronic device provided in an embodiment of the present application;

[0061] FIG11 is a schematic diagram of the input method and voice assistant registration service provided in an embodiment of the present application;

[0062] FIG12 is a signaling diagram of an input interaction method provided in an embodiment of the present application;

[0063] FIG13 is a signaling diagram of another input interaction method provided in an embodiment of the present application. DETAILED DESCRIPTION

[0064] The technical solutions in the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. The terms used in the following embodiments are only for the purpose of describing specific embodiments and are not intended to be limiting of the present application. As used in the specification and appended claims of the present application, the singular expressions "one", "a kind of", "said", "above", "the" and "this" are intended to also include expressions such as "one or more", unless there is a clear contrary indication in the context. It should also be understood that in the embodiments of the present application, "one or more" refers to one, two or more; "and / or" describes the association relationship of associated objects, indicating that three relationships may exist; for example, A and / or B can represent: the existence of A alone, the existence of A and B at the same time, and the existence of B alone, where A and B can be singular or plural. The character " / " generally indicates that the related objects before and after are in an "or" relationship.

[0065] References to "one embodiment" or "some embodiments" in this specification mean that a particular feature, structure, or characteristic described in conjunction with that embodiment is included in one or more embodiments of the present application. Thus, phrases such as "in one embodiment," "in some embodiments," "in other embodiments," and "in yet other embodiments" appearing in various places in this specification do not necessarily refer to the same embodiment, but rather mean "one or more but not all embodiments," unless otherwise specifically emphasized. The terms "including," "comprising," "having," and variations thereof mean "including but not limited to," unless otherwise specifically emphasized.

[0066] The "multiple" involved in the embodiments of the present application means greater than or equal to two. It should be noted that in the description of the embodiments of the present application, the words "first" and "second" are only used for the purpose of distinguishing the description and cannot be understood as indicating or implying relative importance or order.

[0067] Before introducing the embodiments of this application, we first explain some of the terms or concepts involved in the embodiments of this application. It should be understood that this application does not specifically limit the naming of the following terms. The following terms may have other names. The renamed terms still meet the following relevant term explanations.

[0068] 1. Breath Wake-Up: An electronic device recognizes the characteristics of a user speaking close to a microphone. The electronic device can determine whether the user is close to the microphone by detecting whether the user lifts the device and inputs a voice message. The breath wake-up feature can be used by multiple applications on the electronic device, and these applications can use the breath wake-up feature's results to perform tasks.

[0069] 2. Breath awakening voice assistant: The electronic device recognizes the above-mentioned breath awakening characteristics, thereby waking up the voice assistant to respond to the voice interaction command. This replaces the method of waking up the voice assistant when recognizing the user's call for special words in the related art, making it easier for users to use the voice assistant. After the electronic device recognizes the characteristics generated when the user speaks close to the microphone, the electronic device can automatically wake up the voice assistant, and the voice assistant (such as YOYO) will conduct voice interaction with the user. For example, the user only needs to lift the electronic device and say a voice interaction command (which may not be a special command to wake up the voice assistant), and the electronic device can respond to the voice interaction command, without the user having to say a voice command to wake up the voice assistant before saying the voice interaction command.

[0070] 3. Breath-activated speech-to-text: Electronic devices recognize the characteristics of breath-activated speech and activate the input method's speech-to-text service. In specific scenarios, the electronic device recognizes the characteristics of the user speaking close to the microphone and automatically activates the input method's speech-to-text function. For example, when the electronic device's input method is activated, the user lifts the electronic device and speaks. The electronic device then converts the user's voice input into text and displays it in the input method's text input box.

[0071] 4. Eye tracking function: It can be understood as the electronic device tracking the location of the user's gaze point, and by tracking the location of the gaze point, determining whether to expand the banner notification and / or open the application corresponding to the banner notification.

[0072] Figure 1 illustrates a user holding a phone in one hand. As shown in Figure 1(a), the user holds the phone in one hand, with the thumb touching area 101. When the user holds the phone closer to the top, the thumb's touch area, as shown in Figure 1(b), is in area 102. This demonstrates that when a user holds the phone in one hand, it's difficult for the thumb to touch both the top and bottom of the phone.

[0073] In one application scenario, as shown in Figure 2(a), a chat banner notification 201 for an instant messaging application is displayed at the top of a mobile phone. Upon receiving a user click on any location within the chat banner notification, the mobile phone displays the application interface corresponding to the chat banner notification, as shown in Figure 2(b). The application interface includes a toggle button 202 and a text input box 203. When the user clicks the toggle button 202, the mobile phone switches from text input to voice input in response to the user's click. The text input box 203 is used for the user to enter text information. When the user clicks the text input box 203, as shown in Figure 2(b), the mobile phone launches the input method and displays the interface shown in Figure 2(c). The interface includes an input method keyboard, which includes a speech-to-text button 204. When the user long-presses the speech-to-text button 204, the electronic device's input method enters speech-to-text mode. The electronic device then captures the user's voice input and converts it into text, which is then displayed in the text input box 203. The text display box 203 can be understood as the text display area of ​​the input method application.

[0074] In the application scenario shown in Figure 2, a chat banner notification 201 is located at the top of the phone, while a toggle button 202 and a speech-to-text button 204 are located at the bottom of the phone. If a user holds the phone in one hand and clicks the chat banner notification 201 with their thumb, they will not be able to click the toggle button 202 and the speech-to-text button 204, which is inconvenient for the user.

[0075] Based on this, an embodiment of the present application provides an input interaction method that can use breath awakening to automatically activate the voice input to text function, collect the user's input voice, and convert it into text without the user having to manually activate the voice input to text function.

[0076] Taking a mobile phone as an example, Figures 3A to 3E show the application scenarios of using the breath wake-up function provided in the embodiments of the present application.

[0077] FIG3A shows a scenario in which an input method uses breath-activated wakeup to implement the voice input-to-text conversion function, that is, a breath-activated voice-to-text conversion scenario.

[0078] As shown in Figure 3A (a), a chat banner notification 201 of an instant messaging application is displayed at the top of the mobile phone. When the mobile phone receives a click operation from the user on any position in the chat banner notification, the mobile phone displays the application interface corresponding to the chat banner notification as shown in Figure 3A (b). As shown in Figure 3A (b), the user clicks on the text input box 203. In response to the user's click operation, the mobile phone pulls up the input method and displays the interface shown in Figure 3A (c). The user lifts the mobile phone and inputs a voice. The voice-to-text function of the mobile phone is activated. The mobile phone collects the voice input by the user, converts the voice into the corresponding text, and displays the converted text in the text input box as shown in Figure 3A (e).

[0079] In another possible implementation, the user does not need to click on the text input box 203. When the mobile phone displays the interface shown in Figure 3A (b), the user lifts the mobile phone and enters a voice message, and the voice-to-text function of the mobile phone's input method is activated. The mobile phone directly displays the interface shown in Figure 3A (e), and the user's voice input is converted into text and displayed in the text input box. Alternatively, the mobile phone can first display the interface shown in Figure 3A (c) and then display the interface shown in Figure 3A (e), and the user's voice input is converted into text and displayed in the text input box.

[0080] In another possible embodiment, the user does not click on the text input box 203. It can be understood that the input method is not launched and running. When the mobile phone displays the interface shown in Figure 3A (b), the user lifts the mobile phone and inputs a voice message. The breath-activated voice assistant function of the mobile phone is activated, and the voice assistant is enabled to run in response to the voice interaction instructions indicated by the user's voice input, and the text is displayed in the upper layer of the interface shown in Figure 3A (b). For example, Figure 3E (b) below shows a method for displaying text in the upper layer of an interface.

[0081] In another possible embodiment, the user lifts up the phone and inputs a voice, the breath-activated voice-to-text function of the phone is activated, the phone collects the voice input by the user, converts the voice into corresponding text, and displays the converted text in the text input box as shown in (e) in Figure 3A. After that, the user is silent for a period of time, and the duration of this period reaches a threshold. The phone then determines that the user is no longer inputting a voice, and the breath-activated voice-to-text function ends. Moreover, if the user is silent for at least the threshold time and then inputs a voice again, the phone will not convert the user's input voice into corresponding text, and will not display the text in the text input box. In some embodiments, the user lifts up the phone again and inputs a voice, the breath-activated voice-to-text function of the phone is activated, the phone collects the user's input voice, converts the voice into corresponding text, and displays it in the text input box.

[0082] In another possible embodiment, after the breath-awakened voice-to-text function of the mobile phone is activated, the mobile phone collects the voice input by the user and converts the voice into the corresponding text. In the process, the user inputs the operation of closing the input method application. For example, the user clicks on the interface of the instant messaging application in (e) in Figure 3A, or the user inputs the operation of exiting the instant messaging application in the interface shown in (e) in Figure 3A, or the user inputs the operation of opening other applications in (e) in Figure 3A, or the user opens other applications through air gestures to close the input method application. In response to the user's operation, the mobile phone controls the input method application to exit, and the breath-awakened voice-to-text function also ends. The user lifts the mobile phone and inputs voice, but the mobile phone will not convert the voice input by the user into the corresponding text, and will not display the text in the text input box.

[0083] FIG3B shows another scenario of breath-awakened speech-to-text conversion.

[0084] In some application scenarios, the user presses the voice-to-text button 204 as shown in FIG3B (a) to input a voice. The mobile phone responds to the user's operation and activates the voice-to-text function. The input method converts the user's input voice into corresponding text and displays it in the text input box, as shown in FIG3B (b) for example. After the user uses the voice-to-text button 204 to input a paragraph of text, the user raises the mobile phone to the mouth and inputs a voice as shown in FIG3B (c), and the mobile phone activates the breath-activated voice-to-text function. Alternatively, the user clicks on the interface of the instant messaging application as shown in FIG3B (b), and the mobile phone displays the interface as shown in FIG3B (e). The user then raises the mobile phone to the mouth and inputs a voice as shown in FIG3B (c), and the mobile phone activates the breath-activated voice-to-text function. The input method converts the user's input voice into corresponding text and displays the converted text in the text input box as shown in FIG3B (d).

[0085] In another possible implementation, the user clicks "Send" as shown in Figure 3B (b), and the phone displays the interface shown in Figure 3A (c). Of course, the area of ​​the interface displaying the user's conversation should also display "Hello" in the text input box. At this point, the user raises the phone and enters a voice message, which activates the breath-activated voice-to-text function of the phone's input method. The phone displays the interface shown in Figure 3A (e), and the user's voice message is converted into text and displayed in the text input box.

[0086] Figure 3C shows another scenario of breath-awakened speech-to-text conversion.

[0087] In some application scenarios, it is difficult for users to click on the banner notification displayed at the top of the display. To this end, the mobile phone can use the eye tracking function to automatically open the application corresponding to the banner notification without the user having to perform a click operation. In some embodiments, the mobile phone receives the banner notification, activates the eye tracking function, and opens the application corresponding to the banner notification when it detects that the gaze point is in the area where the banner notification is located. Alternatively, the mobile phone opens the application corresponding to the banner notification when it detects that the gaze point has been in the area where the banner notification is located for a preset period of time. When the mobile phone opens the application corresponding to the banner notification and displays the application interface, it directly launches the input method to run. That is, when the mobile phone detects that the user's gaze point has been in the area where the banner notification is located for a preset period of time, it opens the application corresponding to the banner notification and directly displays the interface shown in (c) in Figure 3C. The lower half of the interface displays the content corresponding to the input method. While the application corresponding to the banner notification is running, if the user lifts the phone and enters voice, the phone can also use the breath-activated voice-to-text function to convert the user's input into text and display it in the text input box.

[0088] As shown in (a) of Figure 3C , a chat banner notification 201 of an instant messaging application is displayed on the top of the mobile phone, and the eye tracking function of the mobile phone is turned on. The chat banner notification 201 includes a gaze cursor 701, which indicates that the user's gaze point is located in the area where the banner notification is located. The mobile phone automatically enters the interface of the instant messaging application corresponding to the chat banner notification 201, that is, as shown in (b) of 3C , the interface of the instant messaging application is displayed. In some embodiments, the mobile phone will automatically launch the input method as shown in (c) of Figure 3C . In another embodiment, the mobile phone will not automatically launch the input method. The user can click on the text input box 203 as shown in (b) of Figure 3C . In response to the user's click operation, the mobile phone launches the input method and displays the interface shown in (c) of Figure 3C .

[0089] The phone's input method is activated. The user lifts the phone and enters a voice message. As shown in Figure 3C (d), the phone's breath-activated speech-to-text feature is activated. The phone captures the user's voice input, converts it into text, and displays the converted text in the text input box, as shown in Figure 3C (e).

[0090] In another possible implementation, the user does not need to click on the text input box 203. When the phone displays the interface shown in FIG3C (c), the user raises the phone and enters a voice message. The breath-activated voice-to-text function of the phone's input method is activated. The phone directly displays the interface shown in FIG3C (e), and the user's voice message is converted to text and displayed in the text input box.

[0091] In the application scenario shown in Figure 3C, the mobile phone automatically launches the input method or the user clicks on the text input box to launch the input method. After the mobile phone displays the interface shown in (c) in 3C, the user can also use the method shown in Figure 3B, by pressing the voice-to-text button 204 and starting the breath-awakened voice-to-text function. The input method converts the user's input voice into corresponding text, and displays the converted text in the text input box as shown in (b) and (d) in Figure 3B.

[0092] It should be noted that a chat banner notification 201 is displayed at the top of the mobile phone, and the chat banner notification 201 includes a gaze cursor 701. The mobile phone can also directly enter the instant messaging application corresponding to the chat banner notification 201, and simultaneously start the input method as shown in (a) and (b) in Figure 3D.

[0093] In some embodiments, the phone detects a banner notification, activates eye tracking, and opens the app corresponding to the banner notification when it detects the gaze point is within the banner notification area. While the app corresponding to the banner notification is running, the phone launches an input method and converts the user's voice input into text, which is displayed in the input method's text input box without the user having to lift the phone.

[0094] It should also be noted that in the scenarios shown in Figures 3A to 3C, compared with the voice-to-text function corresponding to the voice-to-text button, the mobile phone uses a breath-awakened voice-to-text function to convert the user's input voice into the corresponding text, which has a strong anti-noise ability and can shield the interference signals in the user's voice input process to a large extent.

[0095] Figure 3E shows a scenario where a voice assistant uses breath wake-up to respond to voice interaction commands, that is, a scenario where a voice assistant is woken up by breath.

[0096] As shown in Figure 3E (a), the phone is in the off-screen state. The user lifts the phone and enters a voice interaction command, such as a voice interaction command to query the weather. The phone activates the voice assistant in response to the voice interaction command, and as shown in Figure 3D (b), the display turns on to display the lock screen interface, and the weather information is displayed on the lock screen interface.

[0097] The breath-awakening speech-to-text function in the two application scenarios shown in Figures 3A to 3D can be manually turned on or off by the user. The following describes several entrances to the breath-awakening speech-to-text function in conjunction with Figures 4 to 8.

[0098] In some embodiments, when a user first uses the voice-to-text function of an input method to input text, the user may be prompted to enable the breath-activated voice-to-text function. After the electronic device enables the breath-activated voice-to-text function in response to the user's operation, the input method of the mobile phone is launched again. The mobile phone can automatically activate the voice-to-text function of the input method using the breath-activated voice-to-text function to collect the user's input and convert it into text.

[0099] Exemplarily, as shown in FIG4 (a), the user long presses the voice-to-text button 204 for the first time in the interface of the instant messaging application. In response to the user's long press operation, the mobile phone can display a pop-up window 205 to remind the user to turn on the breath-awakened voice-to-text function. The pop-up window 205 can be displayed on the upper layer of the interface of the real-time messaging application. The pop-up window 205 can be a toast pop-up window. Exemplarily, as shown in FIG4 (b), the pop-up window 205 includes a prompt "OK to turn on the breath-awakened voice-to-text function", a cancel button and an OK button. The user can click the cancel button to ignore the pop-up window 205, and the user can click the OK button to turn on the breath-awakened voice-to-text function. The user clicks the cancel button, the pop-up window 205 disappears, and the user can manually operate the voice-to-text button 204 to input voice.

[0100] As shown in Figure 4(b), the user clicks the OK button. In response to the user's click, the phone displays a prompt window 206. Prompt window 206 can be displayed on the upper layer of the instant messaging application interface and can be a toast pop-up window. As shown in Figure 4(c), prompt window 206 includes instructions for using the breath-activated speech-to-text function, such as "Tip: Breath-activated speech-to-text is enabled. After positioning the cursor, aim at the microphone at the bottom of the device to start voice input." In this way, the breath-activated speech-to-text function is enabled.

[0101] As another example, as shown in Figure 5(a), the user long-presses the voice-to-text button 204 for the first time on the note interface. In response to the user's click, the phone may also display a pop-up window 205 to remind the user to enable the breath-activated voice-to-text function. The pop-up window 205 may be displayed on the upper layer of the note interface, for example, on the input method keyboard. The pop-up window 205 may also be a toast pop-up window. As shown in Figure 5(b), the pop-up window 205 includes the prompt "Confirm to enable the breath-activated voice-to-text function", a cancel button, and an confirm button.

[0102] As shown in Figure 5(b), the user clicks the OK button. In response to the user's click, the phone displays a prompt window 206. Prompt window 206 can be displayed on the upper layer of the note interface and can be a toast pop-up window. As shown in Figure 5(c), prompt window 206 includes instructions for using the breath-activated speech-to-text function, such as "Tip: Breath-activated speech-to-text is enabled. After positioning the cursor, aim at the microphone at the bottom of the device to start voice input." In this way, the breath-activated speech-to-text function can also be enabled.

[0103] The "first time" mentioned in the above content can be understood as: the first long press of the voice-to-text button 204 in the following situations: 1. The new mobile phone includes the breath-awakened voice-to-text function of the input method, or 2. The software system of the old mobile phone is upgraded, and the upgraded system version includes the breath-awakened voice-to-text function of the input method.

[0104] In other embodiments, users can also enable the breath-activated speech-to-text feature in the input method's settings interface. For example, the input method's input settings or interface settings can include a control for enabling the breath-activated speech-to-text feature. Figures 6 and 7 illustrate implementations of displaying a control for enabling the breath-activated speech-to-text feature in the input method's settings interface.

[0105] Figure 6 shows adding the breath-awakened speech-to-text function item in the input settings interface of the input method.

[0106] As shown in FIG6(a), the mobile phone displays the system and update interface of the settings application, and the system and update interface includes the language and input method item 501. It should be noted that the interface of the settings application of the mobile phone may include the system and update item, and the user can display the system and update interface shown in FIG6(a) by clicking the system and update item.

[0107] The user clicks on the language and input method item 501 on the system and update interface. In response to the user's click, the phone displays the language and input method interface, as shown in Figure 6 (b). The language and input method interface includes the XX input method item 502, and the user clicks on the XX input method item 502. In response to the user's click, the phone displays the XX input method interface, as shown in Figure 6 (c). The XX input method interface includes the input settings item 503, and the user clicks on the input settings item 503. In response to the user's click, the phone displays the input settings interface, as shown in Figure 6 (d). The input settings interface includes the breath wake-up voice-to-text item 504. The breath wake-up voice-to-text item 504 has a sub-interface. The user clicks on the breath wake-up voice-to-text item 504, and the phone displays the breath wake-up voice-to-text sub-interface, as shown in Figure 6 (e), in response to the user's click. Exemplarily, the sub-interface includes a breath wake-up voice-to-text guidance animation and a button 505. The breath awakening voice-to-text guidance animation can be understood as a short video, which can explain the operation instructions of the breath awakening function. A breath awakening voice-to-text button 505 can be provided to the user, and the user can turn on or off the breath awakening voice-to-text function by clicking the breath awakening voice-to-text button 505.

[0108] Figure 7 shows the button for starting the breath-awakened speech-to-text function in the input settings or interface settings of the input method.

[0109] FIG7 (a) shows the interface of the XX input method. FIG7 (a) and FIG6 (c) may be the same interface. The interface of the XX input method includes an input setting item 503 and an interface setting item 504. As shown in FIG7 (a), the user clicks the input setting item 503, and the mobile phone responds to the user's click operation and displays the input setting interface. As shown in FIG7 (b), the input setting interface includes a breath-wake-up voice-to-text button 507, which allows the user to turn on or off the breath-wake-up voice-to-text function by clicking the button.

[0110] As shown in Figure 7 (b), the user clicks the breath wake-up voice-to-text button 507, and the mobile phone responds to the user's click operation and turns on the breath wake-up voice-to-text function. In some embodiments, the mobile phone may also display a pop-up window to remind the user that the breath wake-up function is turned on. For example, Figure 7 (b) shows a pop-up window 508, which is displayed on the upper layer of the input settings interface and includes a reminder that "Breath wake-up voice-to-text is turned on." The pop-up window 508 can also be a toast pop-up window.

[0111] Alternatively, as shown in (a) of FIG7 , the user clicks on the interface setting item 506, and the mobile phone displays the interface setting interface in response to the user's click operation, as shown in (c) of FIG7 , the interface setting interface includes a breath-awakening speech-to-text button 509, which allows the user to turn on or off the breath-awakening speech-to-text function by clicking the button.

[0112] As shown in Figure 7 (c), the user clicks the breath-awakened voice-to-text button 509, and the mobile phone responds to the user's click operation and turns on the breath-awakened voice-to-text function. In some embodiments, the mobile phone may also display a pop-up window to remind the user that the breath-awakened voice-to-text function is turned on. For example, Figure 7 (c) shows a pop-up window 510, which is displayed on the upper layer of the interface setting interface, including the reminder "Breath-awakened voice-to-text is turned on". The pop-up window 510 can also be a toast pop-up window.

[0113] The input method settings interface is not limited to the access method shown in Figure 6. Figure 8 shows another way to access the input method settings interface. After the input method is launched, the input method keyboard display interface includes an input method settings quick launch button, and the user can also enter the input method settings interface through the input method settings quick launch button.

[0114] For example, Figure 8 shows the input method settings interface that opens in the instant messaging application and the note-taking application. As shown in Figure 8 (a), the instant messaging application interface includes an input method settings quick-launch button 601. The user clicks the settings quick-launch button 601, and the phone responds to the user's click by displaying buttons for multiple input method settings on the instant messaging application interface, as shown in Figure 8 (b). Among them, the more settings button 602 is generally the entry button for the input method settings interface. The user clicks the more settings button 602, and the phone responds to the user's click by displaying the XX input method settings interface as shown in Figure 8 (e).

[0115] Similarly, as shown in Figure 8(c), the note interface includes an input method settings quick-launch button 601. When a user clicks on the settings quick-launch button 601, the phone responds by displaying buttons for multiple input method settings on the note interface, as shown in Figure 8(d). Alternatively, when a user clicks on the more settings button 602, the phone responds by displaying the XX input method settings interface, as shown in Figure 8(e).

[0116] The above content uses a mobile phone as an example to introduce the entry point of the breath-activated voice-to-text function. After the breath-activated voice-to-text function is enabled, in application scenarios where the mobile phone opens the input method and needs to enter text, the mobile phone can execute the input interaction method provided in the embodiments of this application to collect the user's audio data and convert it into text for display.

[0117] Of course, the input interaction method provided in the embodiments of the present application can also be applied to electronic devices such as tablet computers, personal digital assistants (PDAs), desktop computers, laptop computers, notebook computers, ultra-mobile personal computers (UMPCs), handheld computers, netbooks, and wearable devices.

[0118] Taking a mobile phone as an example, Figure 9 shows an example of the composition of an electronic device provided in an embodiment of the present application. As shown in Figure 9, the electronic device 100 may include a processor 110, an internal memory 120, a camera 130, a display 140, a mobile communication module 150, a wireless communication module 160, an audio module 170, and a sensor module 180.

[0119] It should be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 100. In other embodiments, the electronic device 100 may include more or fewer components than shown, or may combine or separate certain components, or arrange the components differently. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0120] The processor 110 may include one or more processing units. For example, the processor 110 may include an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, a video codec, a digital signal processor (DSP), a baseband processor, a smart sensor hub, and / or a neural-network processing unit (NPU). The different processing units may be independent devices or integrated into one or more processors.

[0121] Processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in processor 110 is a cache memory. This memory can store instructions or data that have just been used or are being recycled by processor 110. If processor 110 needs to use the same instruction or data again, it can directly access the memory. This avoids duplicate accesses, reduces processor 110 latency, and thus improves system efficiency.

[0122] The internal memory 120 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120. The internal memory 120 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 120 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 120, and / or the instructions stored in the memory provided in the processor.

[0123] In some embodiments, the internal memory 120 stores instructions for executing the input interaction method. The processor 110 can execute the instructions stored in the internal memory 120 to detect that the electronic device meets the conditions for the breath-activated speech-to-text function of the input method, collect the user's audio data, and convert it into text for display.

[0124] The electronic device implements display functionality through a GPU, display screen 140, and an application processor. A GPU is a microprocessor for image processing that connects display screen 140 and the application processor. The GPU performs mathematical and geometric calculations for image rendering. Processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0125] Display screen 140 is used to display images, video interfaces, and the like. Display screen 140 includes a display panel. The display panel can be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a MiniLED, a MicroLED, a Micro-OLED, or a quantum dot light-emitting diode (QLED). In some embodiments, the electronic device can include one or N display screens 140, where N is a positive integer greater than one.

[0126] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 150, the wireless communication module 160, the modem processor and the baseband processor.

[0127] Antenna 1 and Antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve antenna utilization. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In other embodiments, the antennas can be used in conjunction with a tuning switch.

[0128] The mobile communication module 150 can provide solutions for wireless communications including 2G / 3G / 4G / 5G applied to the electronic device 100. The mobile communication module 150 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves from the antenna 1, and filter, amplify, and process the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 110 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the processor 110. In some embodiments, at least some of the functional modules of the mobile communication module 150 can be set in the same device as at least some of the modules of the processor 110.

[0129] The wireless communication module 160 can provide wireless communication solutions including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 160 can be one or more devices that integrate at least one communication processing module. The wireless communication module 160 receives electromagnetic waves via the antenna 2, frequency modulates and filters the electromagnetic wave signals, and sends the processed signals to the processor 110. The wireless communication module 150 can also receive the signal to be sent from the processor 110, frequency modulate it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0130] The electronic device can implement audio functions such as music playback and recording through the audio module 170, the speaker 170A, the receiver 170B, the microphone 170C, the headphone jack 170D, and the application processor.

[0131] The audio module 170 is used to convert digital audio information into analog audio signal output, and is also used to convert analog audio input into digital audio signals. The audio module 170 can also be used to encode and decode audio signals. In some embodiments, the audio module 170 can be provided in the processor 110, or some functional modules of the audio module 170 can be provided in the processor 110.

[0132] The speaker 170A, also called a "speaker," is used to convert audio electrical signals into sound signals. The electronic device can listen to music or make hands-free calls through the speaker 170A.

[0133] The receiver 170B, also called a "handset", is used to convert audio electrical signals into sound signals. When the electronic device receives a call or voice message, the voice can be heard by placing the receiver 170B close to the human ear.

[0134] Microphone 170C, also known as "microphone" or "microphone", is used to convert sound signals into electrical signals. When making a call or sending a voice message, the user can speak by putting their mouth close to the microphone 170C to input the sound signal into the microphone 170C. The electronic device can be provided with at least one microphone 170C. In other embodiments, the electronic device can be provided with two microphones 170C, which can not only collect sound signals but also realize noise reduction function. In other embodiments, the electronic device can also be provided with three, four or more microphones 170C to realize sound signal collection, noise reduction, and identification of sound sources, and realize directional recording function, etc.

[0135] In some embodiments, the at least one microphone 170C provided in the electronic device is located at the bottom of the electronic device and is used to collect voice signals input by the user.

[0136] The headphone jack 170D is used to connect a wired headphone and can be a USB interface, or a 3.5mm open mobile terminal platform (OMTP) standard interface or a cellular telecommunications industry association of the USA (CTIA) standard interface.

[0137] In the sensor module 180, the pressure sensor 180A is used to sense the pressure signal and convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 180A can be set on the display screen 140. There are many types of pressure sensors 180A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a device including at least two parallel plates with conductive material. When force acts on the pressure sensor 180A, the capacitance between the electrodes changes. The electronic device determines the intensity of the pressure based on the change in capacitance. When a touch operation is applied to the display screen 140, the electronic device detects the intensity of the touch operation based on the pressure sensor 180A. The electronic device can also calculate the position of the touch based on the detection signal of the pressure sensor 180A.

[0138] Touch sensor 180B, also known as a "touch-sensitive device," can be disposed on display screen 140. Touch sensor 180B and display screen 140 form a touch screen, also known as a "touch screen." Touch sensor 180B is configured to detect touch operations applied thereto or in the vicinity thereof. The touch sensor can transmit the detected touch operations to an application processor to determine the type of touch event. Visual output related to the touch operations can be provided via display screen 140. In other embodiments, touch sensor 180B can also be disposed on the surface of the electronic device, at a location different from that of display screen 140.

[0139] The acceleration sensor 180C can detect the magnitude of the acceleration of the electronic device in various directions (generally three axes). When the electronic device is stationary, it can detect the magnitude and direction of gravity and can also be used to identify the posture of the electronic device.

[0140] The gyro sensor 180D can be used to determine the motion posture of the electronic device. In some embodiments, the angular velocity of the electronic device around three axes (ie, x, y, and z axes) can be determined by the gyro sensor 180D.

[0141] In addition, the electronic device runs an operating system on top of the above components. operating system, operating system, Operating system, etc. Applications can be installed and run on the operating system.

[0142] FIG10 is a schematic diagram of the software structure of the electronic device provided in an embodiment of the present application.

[0143] The layered architecture divides the operating system of an electronic device into several layers, each with a clear role and division of labor. Layers communicate with each other via software interfaces. In some embodiments, the operating system of the electronic device is the Android system. The Android system can be divided into five layers: from top to bottom: the application (APP) layer, the application framework layer (FWK), the system library, the hardware abstraction layer (HAL), and the kernel layer.

[0144] The application layer can include a series of application packages. As shown in Figure 10, the application package can include applications such as instant messaging applications, notes, breath wake-up engines, input methods, voice assistants, and perception modules.

[0145] The application framework layer provides an application programming interface (API) and programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0146] As shown in FIG10 , the application framework layer may include a window management service, a content provider, a phone manager, a view system, a resource manager, a speech recognition framework (HwSoundTrigger), a multimodal interaction framework, and an input manager (InputManager), etc.

[0147] The window management service is used to manage window programs. The window management service can implement window addition, deletion, display and hiding controls, etc. The content provider is used to store and obtain data and make this data accessible to applications. The data can include video, images, audio, dialed and received calls, browsing history and bookmarks, phone books, etc. The phone manager is used to provide communication functions for electronic devices. For example, the management of call status (including answering, hanging up, etc.). The view system includes visual controls, such as controls for displaying text, controls for displaying pictures, etc. The view system can be used to build applications. The resource manager provides various resources for applications, such as localized strings, icons, pictures, layout files, video files, etc.

[0148] The speech recognition framework can provide the lower-level modules with instructions sent by the breath wake-up engine, and can also provide the breath wake-up engine with events reported by the lower level.

[0149] The multimodal interaction framework uses the Eagle Eye module to monitor the user's gaze and, if it determines that the user's gaze is in the area of ​​a banner notification, opens the corresponding app interface. If the corresponding app interface is currently open, the multimodal interaction framework can launch the input method and display the input method keyboard in the corresponding app interface.

[0150] The input manager can monitor the input method launch event generated when the input method is launched and run.

[0151] The Android Runtime includes core libraries and a virtual machine. The Android Runtime is responsible for scheduling and management of the Android system. In some embodiments of the present application, an application cold start will be executed within the Android Runtime. The Android Runtime can then obtain the application's optimization file status parameters. The Android Runtime can then use the optimization file status parameters to determine whether the optimization file has become outdated due to a system upgrade and return the determination result to the application management module.

[0152] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0153] The application layer and application framework layer run in a virtual machine. The virtual machine executes Java files in the application layer and application framework layer as binary files. The virtual machine manages object lifecycles, stack management, thread management, security and exception management, and garbage collection.

[0154] The system library may include multiple functional modules, such as a surface manager, media libraries, a 3D graphics processing library (such as OpenGL ES), and a 2D graphics engine (such as SGL).

[0155] The surface manager manages the display subsystem and provides fusion of 2D and 3D layers for multiple applications. The media library supports playback and recording of various common audio and video formats, as well as static image files. The media library supports a variety of audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG. The 3D graphics processing library implements 3D graphics drawing, image rendering, compositing, and layer processing. The 2D graphics engine is the drawing engine for 2D drawing.

[0156] The HAL sits between the kernel layer and the application framework layer, serving as a bridge between them. In some embodiments, the HAL layer includes the speech recognition HAL (sound trigger HAL) and the Hawkeye module. The speech recognition HAL is used to enable or disable the breath wake-up algorithm module and report breath wake-up events to the speech recognition framework. The Hawkeye module is used to monitor the user's gaze.

[0157] The kernel layer is the layer between hardware and software. It includes at least a display driver, a sensor driver, an audio driver, and a breath-activated wake-up algorithm module. In some embodiments, the display driver controls the display screen's image display. The sensor driver controls the operation of multiple sensors, such as accelerometers, gyroscopes, pressure sensors, and touch sensors. The audio driver caches user input audio data, and the breath-activated wake-up algorithm module monitors whether the user raises their hand while holding the phone and inputs audio data.

[0158] Under the above five-layer architecture, the electronic device is also provided with a hardware layer, which may include the aforementioned electronic device hardware components, such as acceleration sensors, gyroscope sensors, pressure sensors, and touch sensors. For example, FIG10 shows a display screen and a camera.

[0159] It should be noted that although the embodiments of the present application are based on The system is used as an example, but the basic principles are also applicable to Electronic devices with other operating systems.

[0160] The technical solutions involved in the following embodiments can all be implemented in electronic devices having the above-mentioned hardware architecture and software architecture.

[0161] As explained in the aforementioned breath wake-up terminology, the breath wake-up function can be provided to voice assistants or input methods. That is, electronic devices can use breath wake-up to implement the breath wake-up voice assistant function, and can also use breath wake-up to implement breath wake-up speech-to-text. However, the breath wake-up function cannot be provided to both voice assistants and input methods at the same time. Therefore, electronic devices need to determine the specific voice input service that the breath wake-up function evokes based on specific business scenario requirements, such as evoked the voice assistant or input method's speech-to-text function based on the business scenario.

[0162] In some embodiments, the electronic device may be configured with multiple switch buttons for the breath wake-up function, with the voice assistant and input method corresponding to one switch button respectively. Of course, other switch buttons may also correspond to other applications of the electronic device, providing the breath wake-up function for other applications. Among the multiple switch buttons for the breath wake-up function, only one switch button may be in the selected state at a time. For example, the unselected switch button is grayed out and cannot be operated, and the switch button in the selected state corresponds to the corresponding application that can use the breath wake-up function. In this way, the user can independently select the application provided by the breath wake-up function by selecting a switch button, which is simple and convenient to operate.

[0163] In other embodiments, the electronic device configures switches for the voice assistant and input method to use the breath wake-up function respectively. The voice assistant configures a switch for the breath wake-up voice assistant function, and the input method configures a switch for the breath wake-up speech-to-text function. The user can independently select the application provided by the breath wake-up function by operating the switch. Figures 4 to 7 show how the input method configures a switch for the breath wake-up speech-to-text function. The voice assistant can also configure a switch for the breath wake-up voice assistant function in the manner of Figure 6, which is not described in detail in this embodiment.

[0164] The user turns on the breath-awakening voice-to-text function by pressing a button, thereby starting the breath-awakening voice-to-text function of the input method. When the electronic device determines that the input method is pulled up to run, the electronic device synchronously executes the breath-awakening function and provides the input method with the running result of the breath-awakening function. If the user's lifting action and voice input are detected, the input method converts the user's input voice into text and displays it in the text input box of the input method. When the user turns off the breath-awakening voice-to-text function by pressing a button, the breath-awakening voice-to-text function of the input method is turned off, the electronic device will not execute the breath-awakening voice-to-text function, and the input method will not convert the user's input voice. For example, Figures 3A and 3B show the user turning on the breath-awakening voice-to-text function by pressing a button. During the operation of the input method, the input method uses the running result of the breath-awakening function to convert the user's input voice into text and displays it in the text input box of the input method.

[0165] Similarly, the user turns on the button of the switch for the breath-awakening voice assistant function to activate the breath-awakening voice assistant function of the voice assistant. The electronic device executes the breath-awakening function and provides the voice assistant with the operating results of the breath-awakening function, and the voice assistant responds to the voice interaction instructions input by the user. The user turns off the button of the switch for the breath-awakening voice assistant function to turn off the breath-awakening voice assistant function of the voice assistant. The electronic device will not execute the breath-awakening voice assistant function, and the voice assistant will not respond to the voice interaction instructions input by the user. For example, Figure 3D shows the user turning on the button of the switch for the breath-awakening voice assistant function. The voice assistant uses the operating results of the breath-awakening function to respond to the voice interaction instructions input by the user to query the weather and display the weather information on the display screen.

[0166] It should be noted that the switch for the breath-activated speech-to-text function configured in the input method and the switch for the breath-activated voice assistant function configured in the voice assistant are mutually exclusive switches. At any given moment, only one of the switches for the breath-activated speech-to-text function and the breath-activated voice assistant function is in the on state. In some embodiments, the switch for the breath-activated speech-to-text function configured in the input method is in the off state, while the switch for the breath-activated voice assistant function configured in the voice assistant can be turned on by the user; when the switch for the breath-activated speech-to-text function configured in the input method is turned on, the switch for the breath-activated voice assistant function configured in the voice assistant is simultaneously turned off.

[0167] In other embodiments, the electronic device configures a switch for the voice assistant and the input method to use the breath wake-up function. The voice assistant configures a switch for the breath wake-up voice assistant function, and the input method configures a switch for the breath wake-up speech-to-text function. The input method may also configure the switch for the breath wake-up speech-to-text function as shown in Figures 4 to 7, and the voice assistant may also configure the switch for the breath wake-up voice assistant function as shown in Figure 6.

[0168] When the user turns on the breath-activated voice assistant function and the breath-activated speech-to-text function, the voice assistant and input method can register their services with the breath-activated engine. The breath-activated engine can activate the breath-activated function based on the registered services and provide the breath-activated function results to the voice assistant or input method based on their service needs. This simplifies user operations and intelligently implements the breath-activated function based on the service needs of the input method and voice assistant.

[0169] Figure 11 shows the schematic diagram of the input method and voice assistant registration business.

[0170] As shown in Figure 11, the breath wake-up engine may include a switch state management module and a scenario perception decision module. The switch state management module is used to manage the application's registered services, and the scenario perception decision module is used to determine the current application scenario. When the user turns on the breath wake-up speech-to-text function of the input method, the input method responds to the user's operation and registers the service with the switch state management module of the breath wake-up engine. Similarly, when the user turns on the breath wake-up voice assistant function of the voice assistant, the voice assistant responds to the user's operation and registers the service with the switch state management module.

[0171] The input method or voice assistant service registering its service with the breath wake-up engine's switch state management module can be understood as the input method or voice assistant service registering to use the breath wake-up function. The user enabling the input method's breath wake-up speech-to-text function includes enabling the input method's breath wake-up speech-to-text function in any of the ways shown in Figures 4 to 7.

[0172] If the switch state management module determines that there is a registered service, the breath awakening path is opened. Of course, if the switch state management module determines that there is no registered service, the breath awakening path is closed.

[0173] The breath wake-up path is opened, and the breath wake-up algorithm module is enabled to run synchronously to detect the user's hand-raising action while holding the mobile phone and whether the user inputs audio data. In some embodiments, the breath wake-up algorithm module can detect the user's hand-raising action while holding the mobile phone through motion data collected by the acceleration sensor and / or the gyroscope sensor, and can collect the user's audio data through the microphone. When the breath wake-up algorithm module detects the user's hand-raising action and collects audio data, a breath wake-up event is generated, and the breath wake-up event is reported to the scene perception decision module through the speech recognition hal and the speech recognition framework. The breath wake-up event is used to indicate that the user is speaking close to the microphone.

[0174] The scene perception decision module performs scene query based on the registered business, that is, queries the status of the application corresponding to the registered business. Furthermore, the scene perception decision module distributes breath wake-up events to the application according to the business priority requirements of the application. The business priority of the application refers to the priority of the application using the breath wake-up function to execute the business. In some embodiments, the business priority of the input method is higher than the priority of the voice assistant business. When the scene perception decision module finds that the input method is pulled up and running, it distributes a breath wake-up event to the input method to wake up the speech-to-text function of the input method. Otherwise, it distributes a breath wake-up event to the voice assistant.

[0175] 12 , the following describes the input interaction method performed by the input method after the input method completes service registration using the principle diagram shown in FIG11 .

[0176] As shown in FIG12 , the input interaction method includes:

[0177] S101. The input manager reports an input method launch event to the scene perception decision module of the breath wake-up engine.

[0178] The input method being launched can be understood as the keyboard shown in Figure 3A (c) being launched, with the cursor in the text input box. In some embodiments, as shown in Figure 10, the input manager of the application framework layer can sense that the input method has been launched and generate an input method launch event. The input manager then reports the input method launch event to the scene perception decision module of the breath wakeup engine.

[0179] S102: The sensor collects motion data.

[0180] The sensor in this step may refer to an acceleration sensor and / or a gyroscope sensor.

[0181] In some embodiments, as shown in Figure 11 above, the input method registers its service with the switch state management module, which then enables the breath wake-up path and enables the breath wake-up algorithm module. The breath wake-up algorithm module can detect the user's hand-raising motion while holding the phone using motion data collected by the accelerometer and / or gyroscope. Based on this, the accelerometer and / or gyroscope are activated to collect motion data.

[0182] In other embodiments, the user activates the breath-activated speech-to-text function by turning on a switch for the breath-activated speech-to-text function. When the user turns on the breath-activated speech-to-text function switch, the breath-activated path is also activated by the breath-activated engine, and the breath-activated algorithm module is enabled to run. The breath-activated algorithm module can also detect the user's hand-raising motion while holding the phone using motion data collected by an accelerometer and / or gyroscope sensor.

[0183] S103: The audio driver caches the audio data.

[0184] As described in the aforementioned step S102, the breath awakening algorithm module is enabled to run. The breath awakening algorithm module can also collect the user's audio data through the microphone, and the audio driver of the electronic device caches the user's audio data.

[0185] The audio driver of the electronic device can cache the audio data of the user input collected within a preset time before the current time, and the cached audio data can be refreshed periodically, and the refresh period is shorter than the preset time. Exemplarily, the preset time can be 6 seconds, and the refresh period can be 1 second. For example, assuming that the current time is 00:15:13, the audio driver caches the audio data from 00:15:08 to 00:15:13. At 00:15:14, the audio driver deletes the audio data from 00:15:08 to 00:15:09 and stores the audio data from 00:15:13 to 00:15:14, that is, the audio driver caches the audio data from 00:15:09 to 00:15:14. The audio data cached by the audio driver can be used to perform audio-to-text conversion.

[0186] The above method of caching audio data is merely an example of an embodiment of the present application, and the present application is not limited thereto.

[0187] It should be noted that there is no specific restriction on the execution order of step S102 and step S103, and they can also be understood as being performed simultaneously.

[0188] S104: The audio driver sends audio data to the breath awakening algorithm module.

[0189] The audio driver can send the cached audio data to the breath wake-up algorithm module.

[0190] S105. The sensor sends motion data to the breath algorithm wake-up module.

[0191] The sensor collects motion data and sends it to the breath calculation wake-up method module. Of course, the sensor in this step can also refer to an acceleration sensor and / or a gyroscope sensor.

[0192] It should be noted that there is no specific limitation on the execution order of step S104 and step S105.

[0193] S106: The breath awakening algorithm module generates a breath awakening event.

[0194] As shown in (c) and (d) of Figure 3A, or (c) and (d) of Figure 3B, in an application scenario where the input method is launched, the user lifts up the phone and inputs audio data. The breath wake-up algorithm module detects that the user has lifted up the phone and input audio data through the audio data sent by the audio driver and the motion data sent by the sensor, and the breath wake-up algorithm module generates a breath wake-up event. In some embodiments, the breath wake-up event generated by the breath wake-up algorithm module may include: a session ID, which is used to obtain the audio data.

[0195] S107. The breath awakening algorithm module reports the breath awakening event to the speech recognition HAL.

[0196] Because the breath awakening path is opened, after the breath awakening algorithm module generates a breath awakening event, it can report the breath awakening event to the speech recognition HAL.

[0197] S108. The speech recognition hal reports the breath awakening event to the speech recognition framework.

[0198] The speech recognition hal receives the breath awakening event reported by the breath awakening algorithm module and reports the breath awakening event to the speech recognition framework.

[0199] S109. The speech recognition framework reports the breath awakening event to the scene perception decision module.

[0200] The speech recognition framework receives the breath awakening event reported by the speech recognition HAL and reports the breath awakening event to the scene perception decision module.

[0201] S110: The scene perception decision module detects that the input method is launched.

[0202] The context-aware decision module queries the status of the application corresponding to the registered service. As shown in step S101, the context-aware decision module receives an input method launch event reported by the input manager. The context-aware decision module detects that the input method has been launched and needs to dispatch a breath wake-up event to the input method to activate the input method's speech-to-text function.

[0203] In other scenarios, such as the scenario shown in Figure 3E, the scene perception decision module detects that the input method has not been pulled up, such as not receiving the input method pull-up event reported by the input manager. The scene perception decision module sends a breath wake-up event to the voice assistant, and the voice assistant is enabled to run to obtain the audio data input by the user and respond to the voice interaction instructions indicated by the audio data.

[0204] It should be noted that Figure 12 shows an execution order of step S109 and step S110. It is known that step S109 and step S110 are not limited to the execution order shown in Figure 12, and can be executed in parallel, or step S110 can be executed first and then step S109.

[0205] S111. The scene perception decision module sends a breath wake-up event to the input method.

[0206] When the scene perception decision module detects that the input method is pulled up and running, it sends a breath wake-up event to the input method.

[0207] In some embodiments, the user activates the breath-wake-up speech-to-text function by turning on the switch of the breath-wake-up speech-to-text function. Steps S109 to S111 may not be executed. The breath-wake-up engine receives the breath-wake-up event reported by the speech recognition framework and sends the breath-wake-up event to the input method.

[0208] S112: The input method starts a speech-to-text function and obtains cached audio data through an audio driver.

[0209] The input method needs to convert the audio data input by the user into text. Therefore, the input method needs to obtain the cached audio data from the audio driver. In some embodiments, the input method receives a breath wake-up event, obtains the session ID carried in the breath wake-up event, specifies the session ID to match the audio driver's path, and obtains the audio data cached by the audio driver.

[0210] S113: The input method converts the audio data into corresponding text.

[0211] In some embodiments, the input method may utilize a speech processing model to convert the acquired audio data into text to obtain the corresponding text. This embodiment does not describe the specific implementation process of converting audio data into text.

[0212] S114: The input method controls the display of text.

[0213] After the input method obtains the text corresponding to the audio data, it can control the display of the text. In some embodiments, the input method controls the text to be displayed in the text input box, as shown in (e) in Figure 3A.

[0214] In this embodiment, the breath wake-up algorithm module detects the user raising their hand while holding the phone and collects user input audio data, generating a breath wake-up event. The scene perception decision module receives the breath wake-up event and detects that the input method has been launched, predicting that the user needs to enter text. The scene perception module then sends the breath wake-up event to the input method, triggering it to obtain the user's audio input data and convert it into text for display. This allows the user to enter text into the phone simply by voice input, without having to manually operate buttons on the display screen. This is simple and convenient operation.

[0215] Instant messaging applications will call input methods during operation, so instant messaging applications can be understood as associated applications of input methods. It can be seen from the contents shown in Figures 3B and 3C that if an associated application of an input method such as an instant messaging application generates a banner notification, and the interface of the application corresponding to the banner notification can display the keyboard of the input method, then the mobile phone uses the eye tracking function to detect that the gaze point is in the area where the banner notification is located, and the mobile phone automatically opens the application corresponding to the banner notification and starts the input method to run. In addition, the breath-awakened speech-to-text function of the input method is started and run. The mobile phone collects the voice input by the user through the breath-awakened speech-to-text function and converts the voice into corresponding text.

[0216] The following describes the input interaction method using the functional modules of the input method and the eye gaze function in conjunction with Figure 13. Of course, in some embodiments, before the input method executes the solution shown in Figure 13, it can also complete the service registration using the principle diagram shown in Figure 11.

[0217] As shown in FIG13 , the input interaction method includes:

[0218] S201. The multimodal interaction framework registers the scene fence with the perception module.

[0219] The multimodal interaction framework can be understood as a functional module of the eye tracking function, located in the application framework layer. After the eye tracking function is enabled, the multimodal interaction framework registers the scene fence with the perception module. In some embodiments, registering the scene fence can be understood as registering to monitor whether to generate a banner notification.

[0220] The perception module can be understood as an application module in the application layer, which is used to monitor the status of electronic devices such as mobile phones, such as monitoring whether the electronic devices generate banner notifications.

[0221] S202: The perception module sends a notification message to the multimodal interaction framework to notify that the scene fence condition is met.

[0222] After the multimodal interaction framework registers the scene fence with the perception module, the perception module can monitor whether the electronic device meets the scene fence conditions. As mentioned in the previous steps, registering the scene fence is to register and monitor whether a banner notification is generated. Therefore, the scene fence condition can be understood as the electronic device generating a banner notification and displaying the banner notification.

[0223] When the perception module detects that the electronic device generates and displays a banner notification, it sends a notification message to the multimodal interaction framework to notify the multimodal interaction framework that the electronic device meets the scene fence condition.

[0224] S203: The multi-mode interaction framework controls the operation of the Hawkeye module.

[0225] When the multimodal interaction framework receives the notification message, it sends a control instruction to the Hawkeye module to control the operation of the Hawkeye module.

[0226] The Eagle Eye module can also be understood as a functional module of the eye tracking function, located in the hardware abstraction layer.

[0227] S204: The Eagle Eye module detects eye movement and gazes at a banner notification.

[0228] The Eagle Eye module runs to detect whether the user's gaze is within a specific area of ​​the display. This specific area can be understood as the area where the banner notification is displayed. If the Eagle Eye module detects that the user's gaze is within the specific area of ​​the display, it indicates that the user is looking at the banner notification; otherwise, the user is not looking at the banner notification.

[0229] If the Eagle Eye module detects an eye gaze banner notification, it can control the multimodal interaction framework to execute step S205; otherwise, it can exit this process. In some embodiments, if the Eagle Eye module detects an eye gaze banner notification, it can send a notification message to the multimodal interaction framework to inform the multimodal interaction framework of the detection result.

[0230] S205: The multi-modal interaction framework launches the input method.

[0231] The multimodal interaction framework pulling up the input method can be understood as the multimodal interaction framework pulling up the input method to run, that is, the keyboard shown in (c) in Figure 3A is pulled up, and the cursor is in the text input box.

[0232] Furthermore, the multimodal interaction framework will also pull up the banner notification to run the corresponding application. Of course, the interface of the application corresponding to the banner notification will also be displayed on the display.

[0233] In some embodiments, before executing this step, the multimodal interaction framework may determine that the user has raised their hand while holding the electronic device and inputting audio data. This allows the multimodal interaction framework to detect the user's intention to input voice and launch the input method. The input method interface is displayed in the corresponding application interface of the banner notification. For example, this is shown in Figure 3B (c).

[0234] S206: The input manager reports the input method launch event to the scene perception decision module of the breath wake-up engine.

[0235] As in the aforementioned step S101, the input method is launched and run, and the input manager of the application framework layer can sense that the input method is launched and run, and generate an input method launch event. The input manager reports the input method launch event to the scene perception decision module of the breath awakening engine to inform the scene perception decision module that the input method is in running state.

[0236] S207: The sensor collects motion data.

[0237] The content of step S207 can refer to the content of the aforementioned step S102 and will not be repeated here.

[0238] S208: The audio driver caches the audio data.

[0239] The content of step S208 can refer to the content of the aforementioned step S103 and will not be repeated here.

[0240] There is no specific limitation on the execution order of step S207 and step S208, which can also be understood as being performed simultaneously.

[0241] S209: The audio driver sends audio data to the breath awakening algorithm module.

[0242] The content of step S209 can refer to the content of the aforementioned step S104 and will not be repeated here.

[0243] S210: The sensor sends motion data to the breath algorithm wake-up module.

[0244] The content of step S210 can refer to the content of the aforementioned step S105 and will not be repeated here.

[0245] It should be noted that there is no specific limitation on the execution order of step S209 and step S210.

[0246] S211. The breath awakening algorithm module generates a breath awakening event.

[0247] The content of step S211 can refer to the content of the aforementioned step S106 and will not be repeated here.

[0248] S212. The breath awakening algorithm module reports the breath awakening event to the scene perception decision module through the speech recognition hal and speech recognition framework.

[0249] In some embodiments, the breath awakening algorithm reports the breath awakening event to the speech recognition HAL, the speech recognition HAL receives the breath awakening event, reports the breath awakening event to the speech recognition framework, and the speech recognition framework receives the breath awakening event and reports the breath awakening event to the scene perception decision module.

[0250] S212: The scene perception decision module detects that the input method is launched.

[0251] The content of step S212 can refer to the content of the aforementioned step S110 and will not be repeated here.

[0252] S213. The scene perception decision module sends a breath awakening event to the input method.

[0253] The content of step S213 can refer to the content of the aforementioned step S111 and will not be repeated here.

[0254] S214: The input method starts the speech-to-text function and obtains the cached audio data through the audio driver.

[0255] The content of step S214 can refer to the content of the aforementioned step S112 and will not be repeated here.

[0256] S215: The input method converts the audio data into corresponding text.

[0257] The content of step S215 can refer to the content of the aforementioned step S113 and will not be repeated here.

[0258] S216: The input method controls the display of text.

[0259] The content of step S216 can refer to the content of the aforementioned step S114 and will not be repeated here.

[0260] In this embodiment, the perception module detects that the electronic device generates and displays a banner notification, and then controls the Eagle Eye module through the multi-modal interaction framework to detect the user's gaze point. When it is detected that the user's gaze point is in the area of ​​the banner notification, it means that the user is looking at the banner notification displayed on the display screen, and there is a need to open the application of the banner notification, and there may also be obstacles to manual interaction with the banner notification. Furthermore, the user holds the electronic device and raises his hand, and inputs voice data, and the multi-modal interaction framework starts the input method to run, simplifying the user operation. Further, the scene perception module sends a breath wake-up event to the input method, triggering the input method to obtain the audio data input by the user, and converting it into text display, so that the user does not need to manually operate the buttons on the display screen, and can input text into the mobile phone only by inputting voice, which is simple and convenient to operate.

[0261] Another embodiment of the present application further provides a computer-readable storage medium, which stores instructions. When the computer-readable storage medium is executed on a computer or a processor, the computer or processor executes one or more steps in any of the above methods.

[0262] The computer-readable storage medium may be a non-temporary computer-readable storage medium, for example, the non-temporary computer-readable storage medium may be a read-only memory (ROM), a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0263] Another embodiment of the present application further provides a computer program product comprising instructions, which, when executed on a computer or processor, causes the computer or processor to execute one or more steps in any of the above methods.

Claims

1. An input interaction method, characterized in that: include: The electronic device displays a first interface, wherein the first interface includes a first text input area; During the process of the electronic device displaying the first interface, when the user lifts the electronic device to the mouth and speaks, the first speech-to-text function is turned on, and the electronic device displays the text corresponding to the user input speech in the first text input area; In response to the user's operation on the first text input area, the electronic device displays a second interface, the second interface comprising: a speech-to-text button and a second text input area; When the electronic device displays the second interface, the second voice-to-text function is turned on when the user presses the voice-to-text button, and when the user inputs voice, the text corresponding to the user input voice is displayed in the second text input area.

2. The input interaction method according to claim 1, characterized in that: When the user lifts the electronic device to his mouth to speak, the user does not operate the electronic device.

3. The input interaction method according to claim 1, characterized in that: The first interface does not include a speech-to-text button, or the first interface does not include an interface of an input method application.

4. The input interaction method according to any one of claims 1 to 3, characterized in that: The first interface also includes an interface of a first application.

5. The input interaction method according to claim 4, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a third interface, where the third interface includes a first banner notification of the first application; The electronic device displays the first interface, including: displaying the first interface in response to a click operation of the user's finger on the first banner notification.

6. The input interaction method according to claim 4, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a third interface, where the third interface includes a first banner notification of the first application; The electronic device displays a first interface, comprising: in a process in which the electronic device displays the third interface, in response to a user gazing at the first banner notification, the electronic device displays the first interface.

7. The input interaction method according to any one of claims 1 to 6, characterized in that: In response to the user's operation on the text input area of ​​the first interface, the electronic device displays the second interface, including: In response to a user clicking operation on the first text input area, the electronic device displays the second interface.

8. The input interaction method according to claim 1, characterized in that: Also includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth and speaks, the electronic device invokes a voice assistant, wherein the fourth interface does not include a text input area.

9. The input interaction method according to any one of claims 1 to 8, characterized in that: The electronic device is provided with a first switch, and the first switch is in an on state.

10. The input interaction method according to any one of claims 1 to 8, characterized in that: The electronic device is provided with a first switch, the first switch is in a closed state, and the method further includes: When the electronic device displays the first interface, when the user lifts the electronic device to his mouth and speaks, the electronic device does not activate the first speech-to-text function.

11. The input interaction method according to claim 8, characterized in that: The electronic device is provided with a second switch, and the second switch is in an on state.

12. The input interaction method according to claim 8, characterized in that: The electronic device is provided with a second switch, the second switch is in a closed state, and the method further includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth to speak, the electronic device does not wake up the voice assistant.

13. An input interaction method, characterized in that: include: The electronic device displays a first interface, wherein the first interface includes a first text input area and a speech-to-text button; During the process of the electronic device displaying the first interface, when the user lifts the electronic device to his mouth to speak, the electronic device turns on the first speech-to-text function, switches the speech-to-text button from a first state to a second state, and displays text corresponding to the user's speech input in the first text input area, wherein the user does not operate the speech-to-text button when the user lifts the electronic device to his mouth to speak.

14. The input interaction method according to claim 13, characterized in that: The first state is a closed state, the second state is a first open state, In the first on state, the electronic device continuously receives voice input from the user, and converts the voice input into text and displays it in the first text input area.

15. The input interaction method according to claim 13, characterized in that: The first state is the second on state, the second state is the first on state, In the second on state, when the electronic device detects that the user clicks the speech-to-text button, it receives the user's speech input and converts the speech input into text and displays it in the first text input area; In the first on state, the electronic device continuously receives voice input from the user, and converts the voice input into text and displays it in the first text input area.

16. The input interaction method according to claim 13, characterized in that: The first text input area of ​​the first interface also displays text.

17. The input interaction method according to claim 13, characterized in that: The first interface also includes an interface of a first application.

18. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In response to a user clicking operation on the first banner notification, displaying the third interface, wherein the third interface includes an interface of the first application and a second text input area; The electronic device displays the first interface including: displaying the first interface in response to a user clicking operation on the second text input area.

19. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In response to a user clicking operation on the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

20. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; The electronic device displays the first interface including: in the process of the electronic device displaying the second interface, in response to a user gazing at the first banner notification, the electronic device displays the first interface.

21. The input interaction method according to claim 20, characterized in that: When the user looks at the first banner notification, the user does not operate the screen of the electronic device.

22. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: displaying the first interface in response to a user clicking operation on the second text input area.

23. The input interaction method according to claim 17, characterized in that: Before the electronic device displays the first interface, the electronic device further includes: The electronic device displays a second interface, where the second interface includes a first banner notification of the first application; In the process of the electronic device displaying the second interface, in response to the user looking at the first banner notification, displaying the third interface, wherein the third interface includes a second text input area; The electronic device displays the first interface including: the electronic device automatically displays the first interface after displaying the third interface, and the user does not operate the second text input area after the electronic device displays the third interface and during the process of automatically displaying the first interface.

24. The input interaction method according to claim 23, characterized in that: When the user looks at the first banner notification, the user does not operate the screen of the electronic device.

25. The input interaction method according to claim 13, characterized in that: Also includes: When the electronic device displays the fourth interface or turns off the screen, when the user lifts the electronic device to his mouth to speak, the electronic device invokes a voice assistant, wherein the fourth interface does not include a text input area.

26. The input interaction method according to any one of claims 13 to 25, characterized in that: The electronic device is provided with a first switch, and the first switch is in an on state.

27. The input interaction method according to any one of claims 13 to 25, characterized in that: The electronic device is provided with a first switch, the first switch is in a closed state, and the method further includes: When the electronic device displays the first interface, when the user lifts the electronic device to his mouth and speaks, the electronic device does not activate the first speech-to-text function.

28. The input interaction method according to claim 25, characterized in that: The electronic device is provided with a second switch, and the second switch is in an on state.

29. An electronic device, characterized in that: include: one or more processors, memory, and display screens; The memory and the display screen are coupled to the one or more processors, the memory is used to store a computer program, the computer program includes computer instructions, and when the one or more processors execute the computer instructions, the electronic device executes the input interaction method as described in any one of claims 1 to 12, or the input interaction method as described in any one of claims 13 to 28.

30. A computer-readable storage medium, characterized in that: Used to store a computer program, which, when executed, is specifically used to implement the input interaction method as described in any one of claims 1 to 12, or the input interaction method as described in any one of claims 13 to 28.

Images