Method for converting voice into text, electronic equipment and readable storage medium

Abstract

The invention provides a method for converting voice into characters, electronic equipment and a readable storage medium, and belongs to the technical field of electronic equipment.The method comprises the steps that an input box is displayed in a first interface of a first application in response to a first operation, and the breath awakening function of the electronic equipment is in an on state; in response to display of the input box, setting a second application to which the input box belongs to the highest priority of using the breath wake-up function; wherein the breath awakening function triggers the second application with the highest priority to continuously receive the voice data when the user moves the electronic equipment and the breath of the user is received; in response to the fact that the user moves the electronic equipment and detects the breath of the user, a second application of the electronic equipment continuously receives the voice data and converts the voice data into characters; and displaying the converted characters in the first interface. According to the method for converting the voice into the text, the operation is simple, natural and smooth, the method is suitable for the scene of one-hand operation, and the user experience can be greatly improved.

Description

Technical Field

[0001] The present application relates to the field of electronic technology, and in particular to a method for converting speech to text, an electronic device and a readable storage medium. Background Art

[0002] With the development of computer technology, speech recognition technology is becoming more and more popular. In human-computer interaction, people can express their ideas to electronic devices conveniently and quickly through voice input. As a result, more and more speech recognition technologies are being applied to various applications. For example, in the scene of chatting and social networking, text input is the most frequent scene, and speech-to-text is the second most common text input method after pinyin input.

[0003] Currently, when converting speech to text, you need to long press the speech-to-text button in the input panel of the application to complete the speech-to-text input. However, in some scenarios, such as holding the phone in one hand, when the input method panel pops up, the user's thumb cannot reach or is inconvenient to press the speech-to-text button for a long time. Summary of the invention

[0004] In view of this, the present invention provides a method for converting speech to text, an electronic device and a readable storage medium, wherein the method can.

[0005] Some embodiments of the present application provide a method for converting speech to text. The present application is introduced from multiple aspects below, and the embodiments and beneficial effects of the following aspects can be referenced to each other.

[0006] In a first aspect, the present invention provides a method for converting speech to text, which can be applied to electronic devices, and the method comprises:

[0007] In response to the first operation, an input box is displayed in the first interface of the first application, wherein the breath wake-up function of the electronic device is turned on; in response to the display of the input box, the second application to which the input box belongs is set to the highest priority for using the breath wake-up function; wherein, when the user moves the electronic device and receives the user's breath, the breath wake-up function triggers the second application with the highest priority to continuously receive voice data; in response to the user moving the electronic device and detecting the user's breath, the second application of the electronic device continuously receives voice data and converts the voice data into text; the electronic device displays the converted text in the first interface.

[0008] According to the method of converting speech to text in the embodiment of the present application, during the process of triggering speech input, the user does not need to press the speech input function key or click the function key, and the speech input can be completed naturally. This input method is simple to operate, natural and smooth, suitable for one-handed operation, and can greatly improve the user experience.

[0009] As an embodiment of the first aspect, the first application is a note application, and the first operation is an operation of creating a new note or an editing operation acting on a note detail page. The operation confirms that the user wants to input information, which serves as a trigger condition for setting the priority, and the action is regular and natural.

[0010] As an embodiment of the first aspect, the first application is an instant messaging application, and the first operation is an operation in a dialog interface of the instant messaging application. The operation confirms that the user wants to input information, which serves as a trigger condition for setting the priority, and the action is conventional and natural.

[0011] As an embodiment of the first aspect, in response to a user moving an electronic device and detecting the user's breath, a second application of the electronic device continuously receives voice data and converts the voice data into text, including: detecting that the movement acceleration of the electronic device reaches a preset value, and detecting the user's breath through a breath wake-up function, the second application of the electronic device continuously receives voice data and converts the voice data into text.

[0012] As an embodiment of the first aspect, the second application to which the input box belongs is set to the highest priority for using the breath wake-up function, including: registering the callback event of the second application for the breath wake-up function in the registration list, and setting the callback priority corresponding to the second application to the highest priority. By setting the priority, it can be ensured that the obtained voice data is sent to the second application, and recorded in the form of a registration list, which is convenient for query and quickly confirms the registrant with the highest priority.

[0013] As an embodiment of the first aspect, the method also includes: in response to the third operation, the electronic device cancels the highest priority of the second application using the breath wake-up function, and no longer sends voice data to the second application when it is received, so that when the voice input of the first application is not used, the voice stream obtained by the breath wake-up algorithm will no longer be transmitted to the first application.

[0014] As an embodiment of the first aspect, the third operation includes: closing the input box; or, exiting the running interface of the second application from the current interface.

[0015] As an embodiment of the first aspect, after the highest priority of the second application using the breath wake-up function is canceled, the highest priority is restored to the third application; in response to the user moving the electronic device and detecting the user's breath, the third application of the electronic device continues to receive voice data, so that when the user does not use voice-to-text, the electronic device can quickly set the priority of the third application, for example, restore the priority of the voice assistant application, so that the second application and the third application can achieve seamless switching when using the breath wake-up function, and the user is more comfortable to use.

[0016] As an embodiment of the first aspect, the third application is a voice assistant application.

[0017] In a second aspect, the present application also provides an electronic device, comprising: a memory for storing instructions executed by one or more processors of the device, and a processor for executing instructions so that the electronic device executes any one of the methods in the technical solution of the first aspect.

[0018] In a third aspect, the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the processor executes any one of the methods in the technical solution of the first aspect.

[0019] A fourth aspect is a chip structure, characterized in that it comprises at least one chip, and the at least one chip is used to execute any one of the methods in the technical solution of the first aspect.

[0020] In a fifth aspect, the present application also provides a computer program product comprising instructions, which, when executed on an electronic device, enables a processor to execute any one of the methods in the technical solution of the first aspect. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] Figure 1 A schematic diagram of interface operations when a user interacts with a mobile phone by voice in some embodiments;

[0022] Figure 2 A scene diagram of a user operating a mobile phone with one hand in some implementation examples;

[0023] Figure 3a This is an operation interface diagram of a user turning on the breath awakening function for the first time according to an embodiment of the present application;

[0024] Figure 3b A schematic diagram of an interface for voice interaction between a user and an electronic device according to an embodiment of the present application

[0025] Figure 4 A schematic diagram of the structure of an electronic device according to an embodiment of the present application;

[0026] Figure 5 A software structure block diagram of an electronic device according to an embodiment of the present application;

[0027] Figure 6 A schematic diagram of data flow of a method for converting speech to text in a software framework of an electronic device according to an embodiment of the present application;

[0028] Figure 7 A data flow diagram of a method for converting speech to text in a software framework of an electronic device according to another embodiment of the present application;

[0029] Figure 8 A flowchart of a method for converting speech to text according to an embodiment of the present application;

[0030] Fig. 9 This is an operation interface diagram of enabling the breath awakening function by setting an embodiment of the present application;

[0031] Fig.10 This is a schematic diagram of the interface for voice input according to an embodiment of the present application;

[0032] Fig.11 This is a schematic diagram of an operation interface corresponding to opening a chat application in a mobile phone according to an embodiment of the present application;

[0033] Fig.12 A flowchart of interactions between various modules of a method for converting speech to text executed in a mobile phone according to an embodiment of the present application;

[0034] Fig.13 A block diagram of a system on a chip according to some embodiments of the present application. DETAILED DESCRIPTION

[0035] The technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application.

[0036] In order to facilitate the understanding of the technical solution of the present application, the technical problem to be solved by the present application is first described.

[0037] refer to Figure 1 , Figure 1 Schematic diagram of interface operation when a user interacts with a mobile phone by voice in some embodiments is shown. Figure 1 As shown in the figure, the scenario of user interaction with mobile phone. When the user wants to input text by voice-to-text conversion, the user needs to open the corresponding application, such as Figure 1 In the input interface 10 of (a) of FIG. 1 , the user can select handwriting, or long press the control 11 to make the mobile phone recognize the user's voice through voice recognition technology and convert it into text. In response to the user's click operation on the control 11, as shown in FIG. Figure 1 As shown in (b) of FIG. 20 , in the interface 20 , the user long presses the control 11 (voice input key) and says “what’s the weather like today”. The mobile phone recognizes the user’s voice and converts it into text, which is displayed in the interface. If the user finishes inputting, the user can release the control 11. Figure 1 As shown in (c), the user releases the control 11, the mobile phone determines that the user input is finished, converts the corresponding voice into text, and the input panel (input box) restores the original input state so that the user can reselect the input method. This voice-to-text method allows users to no longer input by handwriting, and the input method is more convenient and quick.

[0038] However, the above method still requires the user to long press the control 11. In some scenarios, since the user operates the mobile phone with one hand, it is not suitable to press the control 11 for a long time, resulting in inconvenience in operation.

[0039] refer to Figure 2 , Figure 2 Schematic diagrams of some implementation examples showing a user operating a mobile phone with one hand. Figure 2 As shown, when the user operates the mobile phone with one hand, for example, one hand holds a bag and the other hand operates the mobile phone. At this time, if the user wants to use the voice-to-text method, the user needs to hold up the mobile phone with one hand and use the thumb to press the control 11 before voice input. In this case of one-handed operation, for electronic devices with large mobile phone screens, such as Figure 2 In the operation, the control 11 is not easy to be touched by the thumb. Even after touching it, the thumb needs to press the control 11 for a long time, and sometimes the pressing force is not enough, or it slips, etc., causing the user to passively end a voice, which reduces the user's experience.

[0040] In addition, sometimes users prefer to interact with their mobile phones more naturally and smoothly. When inputting, users do not want to consciously press or click the voice input button to achieve voice-to-text conversion. Therefore, the current voice-to-text conversion method cannot meet the needs of users.

[0041] Based on the above problems, the present application provides a method for converting speech to text. When the breath wake-up function of the electronic device is turned on, the user does not need to manually press the voice input button when in the input interface of the application, and does not need to press the voice input button for a long time. Instead, even in the case of one-handed operation, speech-to-text can be triggered through conventional natural movements, so that users can quickly realize speech-to-text. And it can meet the user's demand for natural action operation and better improve the user experience.

[0042] It should be noted that the breath wake-up function in the embodiment of the present application refers to an algorithm that can obtain both the data of the user's mobile electronic device and the user's breath. After the breath wake-up function is turned on, if the user operates the mobile phone, such as lifting the mobile phone and inputting voice, the breath wake-up function recognizes the user's operation and can continuously obtain the user's voice.

[0043] The following describes the method of converting speech to text in accordance with an embodiment of the present application in conjunction with the accompanying drawings.

[0044] In the embodiment of the present application, when using a mobile phone to realize voice-to-text conversion, the breath awakening function can be turned on in advance. For example, when a user uses a mobile phone for the first time, the breath awakening function is turned on in the settings function of the application. Among them, the user can refer to the operation interface of turning on the breath awakening function. Figure 3a The operation interface shown.

[0045] refer to Figure 3a , Figure 3a The user's operation interface diagram of the first time opening the breath awakening function in the embodiment of the present application is shown. Figure 3a As shown in (i), when the user uses the mobile phone for the first time, or when the user uses the input method application for the first time, a guide box 311 will be automatically displayed in the input method application interface 310. For example, it reminds the user to use the breath-awakened voice-to-text function: "After turning it on, lift the phone, put your lips about 5 cm away from the bottom microphone, and start voice input immediately." And the corresponding "Cancel" and "Open Now" options. When the user clicks "Open Now", the function activation interface corresponding to "Breath-awakened voice-to-text" (breath-awakening function) is entered. The user can turn on the breath-awakening function through the switch key 321. After the above-mentioned breath-awakening function is turned on, the user can perform voice-to-text conversion operations at any time according to his or her needs.

[0046] refer to Figure 3b , Figure 3b The following is a schematic diagram of the interface for user-mobile phone voice interaction according to an embodiment of the present application. Figure 3b As shown in (i) in the figure, the note application is opened and the note title interface 310 is displayed. Since the user has already turned on the breath wake-up function, there is no need to manually turn on the breath wake-up function at this time. When the user wants to create a new note, the user can click the new control 331 in the interface 330 (the first operation). The mobile phone receives the user's click operation on the control 331, and in response to the click operation on the interface 330, enters the Figure 3b The mobile phone displays an input box 341 in the interface 340, so that the user can input information in the interface 340 of the first application at any time. When the user picks up the mobile phone and speaks close to the mouth, the mobile phone receives this operation and responds to the operation to enter the following operation: Figure 3b The mobile phone converts the received voice into text and displays it in the interface 350, such as "What's the weather like today". When the user inputs the voice, the letter or symbol control in the input box 341 in the interface 340 disappears, and the corresponding display is the amplitude 351 of the audio. When the user clicks the close control 352, or there is no voice input within 5 seconds, Figure 3b The input box 361 shown in (iv) in FIG. 3 can be restored to its original state, that is, Figure 3bThe input box 341 shown in (ii) in FIG. 1 re-displays controls such as letters or symbols.

[0047] In the above Figure 3b In the described operation process, when the user wants to input by voice-to-text, the user only needs to pick up the phone and speak after the input panel pops up to trigger the voice-to-text function. The user does not need to press the input voice function key or click the voice function key, etc., and the voice input can be completed naturally. The operation is simple, natural and smooth, suitable for one-handed operation scenarios, and can greatly improve the user experience.

[0048] In the above embodiments, the description is made by taking a mobile phone as an example of an electronic device. In some embodiments, the electronic device may also be a tablet computer, an e-reader, a remote controller, a personal computer (PC), a laptop computer, a personal digital assistant (PDA), a vehicle-mounted device, an Internet TV, a wearable device, a television, and other products with a display interface, as well as smart display wearable products such as smart watches and smart bracelets. The embodiments of the present application do not impose any special restrictions on the form of the above-mentioned electronic devices. For the convenience of explanation, the following embodiments are all exemplified by taking the electronic device as a mobile phone as an example.

[0049] The method of converting speech to text in an embodiment of the present application is described below in conjunction with the specific structure of an electronic device.

[0050] Figure 4 1 shows a schematic diagram of the structure of the electronic device 100. The electronic device 100 may include a processor 110, an external memory interface 120, an internal memory 121, a universal serial bus (USB) connector 130, a charging management module 140, a power management module 141, a battery 142, an antenna 1, an antenna 2, a mobile communication module 150, a wireless communication module 160, an audio module 170, a speaker 170A, a receiver 170B, a microphone 170C, an earphone interface 170D, a sensor module 180, a button 190, a motor 191, an indicator 192, a camera 193, a display screen 194, and a subscriber identification module (SIM) card interface 195. The sensor module 180 may include a pressure sensor 180A, a gyroscope sensor 180B, an air pressure sensor 180C, a magnetic sensor 180D, an acceleration sensor 180E, a distance sensor 180F, a proximity light sensor 180G, a fingerprint sensor 180H, a temperature sensor 180J, a touch sensor 180K, an ambient light sensor 180L, a bone conduction sensor 180M, etc.

[0051] It is to be understood that the structure illustrated in the embodiment of the present invention does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or separate some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0052] In some embodiments, after the processor 110 opens an application, such as a note application, and when the breath wake-up function is turned on, if the processor 110 receives a first operation, such as clicking on the note title, or clicking on the new note function key, etc., the input interface corresponding to the note application is displayed on the display screen 194, and the input box is displayed in the input interface. Corresponding to the pop-up input box, the processor 110 sets the input method application to which the input box in the note application belongs to the highest priority for breath wake-up. When the processor 110 receives a second operation, such as holding the electronic device to 5 cm from the mouth and speaking, the processor 110 obtains the voice input by the user, and converts the voice stream corresponding to the voice into text, and displays it in the input interface corresponding to the note application through the display screen 194.

[0053] Among them, the second operation may include detecting that the position change of the electronic device meets the preset conditions and detecting the user's voice breath. For example, it may be an operation in which the user picks up the mobile phone and brings it close to the mouth to speak. Among them, the action of the user picking up the mobile phone can be detected by the acceleration sensor 180E, the movement acceleration of the electronic device 100, and the rotation angle of the electronic device detected by the gyroscope sensor 180B. When these data meet the preset conditions, it can be determined that the user has picked up the mobile phone. In some embodiments, the action of approaching the mouth can detect the distance of the object through the distance sensor 180. In some embodiments, the operation of the user picking up the mobile phone and approaching the mouth can also be identified by the action recognition model. Among them, the judgment of the specific action can refer to the recognition method of the prior art, and this application will not be described in detail.

[0054] In some embodiments of the present application, the timing when the processor 110 controls the input box to pop up can be as described in the above-mentioned note application. When the processor 110 receives the first operation, the input interface is directly displayed on the display screen 194, and the input box is displayed. In other embodiments, the processor 110 can control the display screen 194 to display the input interface (the input box does not pop up at this time) after opening the application, such as when opening the chat application, but only when the user clicks the (first operation) edit box in the input interface, the input method application is controlled to pop up the input box in the input interface. In response to the display of the input box, the processor 110 sets the input method application to which the input box in the corresponding interface of the note application belongs as the highest priority for breath awakening.

[0055] It is understandable that the interface connection relationship between the modules illustrated in the embodiment of the present invention is only a schematic illustration and does not constitute a structural limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may also adopt different interface connection methods in the above embodiments, or a combination of multiple interface connection methods.

[0056] The electronic device 100 implements the display function through a GPU, a display screen 194, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 194 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 110 may include one or more GPUs that execute program instructions to generate or change display information.

[0057] In one embodiment, the display screen 194 displays the input interface of the first application according to the control instructions of the processor 110, as well as displays the interface status during voice input and the interface containing text information after voice input, so that the user can see the text information converted from the voice input through the display screen.

[0058] The internal memory 121 can be used to store computer executable program codes, and the executable program codes include instructions. The internal memory 121 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, at least one application required for a function (such as a chat application, an input method application, a note application, an image playback function, etc.), etc. The data storage area may store data created during the use of the electronic device 100 (such as voice stream data, a phone book, etc.), etc. In addition, the internal memory 121 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc. The processor 110 executes various functional applications and data processing of the electronic device 100 by running instructions stored in the internal memory 121 and / or instructions stored in a memory provided in the processor.

[0059] In one embodiment of the present application, the internal memory 121 may store instructions for a method of converting speech to text. The processor 110 runs the instructions for the method of converting speech to text so that the electronic device 100 converts the speech stream into text after the user inputs speech, and displays it on the display screen 194.

[0060] The touch sensor 180K is also called a "touch control device". The touch sensor 180K can be set on the display screen 194. The touch sensor 180K and the display screen 194 form a touch screen, also called a "touch control screen". The touch sensor 180K is used to detect touch operations acting on or near it, for example, a touch operation of clicking an edit box. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 194. In other embodiments, the touch sensor 180K can also be set on the surface of the electronic device 100, which is different from the position of the display screen 194.

[0061] In some embodiments, when a user touches the display screen 194 and it is regarded as a specific click or other operation, the processor 110 receives the user's specific operation on the input panel (input box) of the display screen 194, and in response to the operation, controls the display screen 194 to display the input panel to facilitate direct text input by the user, or voice-to-text input.

[0062] Microphone 170C, also called "microphone" or "microphone", is used to convert sound signals into electrical signals.

[0063] In some embodiments, the microphone 170C can obtain the breath or voice information input by the user, and convert the sound signal into an electrical signal, and transmit it to the processor 110, which further processes the electrical signals, and finally recognizes the electrical signals through voice recognition technology and obtains corresponding text signals. The electrical signals are then displayed on the display screen 194 in the form of text.

[0064] The software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-core architecture, a micro-service architecture, or a cloud architecture. In the embodiment of the present invention, the Android system of the layered architecture is taken as an example to exemplify the software structure of the electronic device 100.

[0065] Figure 5 1 is a software structure block diagram of the electronic device 100 according to an embodiment of the present invention.

[0066] The layered architecture divides the software into several layers, each with clear roles and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and the system library (not shown), as well as the hardware abstraction layer and the kernel layer.

[0067] The application layer can include a series of application packages.

[0068] like Figure 5As shown, the application package can include many applications, such as chat applications, input method applications, note applications, etc. In addition, it can also include camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, short message, etc.

[0069] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0070] For example, the application framework layer may include a window manager, a content provider, a view system, a telephony manager, a resource manager, a notification manager, and the like.

[0071] like Figure 5 As shown, the application framework layer may also include a sound trigger service, and an input method management service.

[0072] Among them, the Input Method Manager Service (IMMS) is used to receive messages sent from the application layer, for example, a message sent by the input method application to enable the breath wake-up function, or a voice input message corresponding to a specific application.

[0073] In some embodiments, the input method management service receives a message to enable the breath-awakened speech-to-text function, saves the enabled state, and notifies the sound trigger service to prepare to start opening the voice channel.

[0074] In other embodiments, the input method management service receives an operation in which the user wants to input voice, for example, the user clicks on an edit box. The corresponding input method application pops up an input method panel in the corresponding interface in response to the user's click operation. The input method management service obtains the event of the input method panel popping up, and prepares for the input method application to receive voice information, for example, registering for wake-up message reception, registering callback events for the input method application, etc. In addition, the input method management service also sets a higher wake-up priority for the input method application, for example, setting the priority of the current input method application to receive voice to a higher priority than YOYO. And when the input panel is closed, the input method management service can also remove the priority of registering for wake-up message reception and restore the priority in the original list, for example, restoring the priority of YOYO.

[0075] In some embodiments, the input method management service can also manage the registration list, for example, sorting by priority in the queue, or marking the highest priority 1 in the list, the second highest priority 2, etc. And, when voice input is not needed, restore the original priority of the registration list.

[0076] like Figure 5 As shown, the Sound Trigger service (sound trigger and scenario decision) is used to receive voice-related messages sent from the input method management service, and will further notify the hardware abstraction layer and the kernel layer so that the electronic device can always maintain a state of receiving voice streams.

[0077] In some embodiments, the sound trigger service can also receive the electrical signal corresponding to the voice message transmitted from the bottom layer, and determine to send the electrical signal corresponding to the voice message to the corresponding application based on the priority information registered by the input method management service. The sending process can be sent directly by the sound trigger service to the application corresponding to the application layer, such as the input method application or the chat application. The electrical signal corresponding to the voice message can also be sent to the input method management service, which will send it to the application in the application layer based on the priority recorded in the registry.

[0078] In some embodiments, the sound trigger service can also manage the priority of the application sending the message, for example, view, register or modify it.

[0079] In some embodiments, when the electronic device receives a user's operation for inputting voice, for example, the user lifts up the mobile phone and puts it to the mouth to start speaking, the action is recognized and confirmed by the breath awakening algorithm, and the user's voice stream is obtained, and the HAL layer is notified to the upper layer in turn, and the relevant information corresponding to the voice stream, such as the session address (session Id) and the buffer size (buffer Size), is passed to the sound trigger service. The sound trigger service sends the relevant information corresponding to the corresponding voice stream, such as the session Id and the buffer size, to the chat application based on the priority of the corresponding application registered by the input method management service. The chat application converts the voice stream into text information and displays it on the display screen, thereby realizing the process of converting text through voice input by the user.

[0080] like Figure 5 As shown, the Hardware Abstraction Layer (HAL) is located at the interface layer between the operating system kernel and the hardware circuit, and its purpose is to abstract the hardware.

[0081] In some embodiments, the hardware abstraction layer includes a voice trigger driver and a microphone driver. Among them, the voice trigger driver shell triggers the breath wake-up algorithm of the kernel layer to keep ready for reception. The microphone driver can drive the hardware microphone to work and obtain the user's voice. For example, when the microphone driver receives the preparation for receiving the voice stream sent from the application framework layer, it can drive the hardware microphone (micphone) to turn on, and when the user inputs the voice, the audio signal of the voice is converted into an electrical signal, and the breath wake-up algorithm of the kernel layer is further used to identify the specific content of the voice stream, etc.

[0082] like Figure 5 As shown, the kernel layer is a layer between hardware and software, and the kernel layer may include at least a display driver, a camera driver, an audio driver, a sensor driver, etc. The kernel layer may also include a breath awakening algorithm, which may perform voice recognition on the voice acquired by the microphone, or may recognize the user's actions, etc. For example, the breath awakening algorithm may include a motion recognition module and a breath recognition module, wherein the motion recognition module is used to recognize the user's actions, and the data of the user's actions may be acquired through a sensor and sent to the breath awakening algorithm. The breath recognition module is used to recognize the breath of the user when speaking, and the breath corresponding data may be input through a microphone. Thereafter, the breath awakening algorithm continuously acquires the voice stream, and uploads the voice stream to the hardware abstraction layer respectively, and finally transmits the voice stream to the application layer, so that the application corresponding to the application layer can obtain the voice stream and convert the voice stream into text, which is displayed on the display screen.

[0083] In some embodiments of the present application, the action recognition module and the breath recognition module of the breath awakening algorithm can be integrated together or can be two separate modules, which is not limited in the present application.

[0084] Combine the following Figure 5 The software framework layer shown further illustrates the data flow of the speech-to-text method.

[0085] refer to Figure 6 , Figure 6 The following is a schematic diagram showing the data flow of the speech-to-text method in the software framework of the electronic device according to the embodiment of the present application. Figure 6 As shown, the electronic device includes two parts: software and hardware. The software part includes an application layer, an application framework layer, a hardware abstraction layer and a kernel layer.

[0086] like Figure 6 As shown, the application layer includes input method applications. The application framework layer includes input method management service IMMS and sound trigger and scenario strategy (referred to as sound trigger service). The hardware abstraction layer includes sound trigger driver and microphone driver. The kernel layer includes breath wake-up algorithm.

[0087] The following describes the data flow of the speech-to-text method from three aspects: enabling the breath wake-up algorithm, setting application priority, and voice input.

[0088] In the first part, we first describe the process of starting the breath awakening algorithm.

[0089] Before the user inputs voice, the breath awakening algorithm needs to be enabled. Figure 6 As shown, the activation process of the breath awakening algorithm can be referred to Figure 6 In the embodiment of the present application, the user can enable the breath awakening voice to text function (turn on the breath awakening algorithm) through the setting function in the input method application and the input method application setting interface. For example, corresponding to Figure 6 In step 1, the user turns on the breath wake-up function through the settings in the input method application. For example, after clicking the function key to turn on the breath wake-up function, the setting module sends the corresponding notification of turning on the breath wake-up algorithm to IMMS. Step 2, IMMS receives the notification of turning on the breath wake-up algorithm, manages the notification, records and saves it, and notifies the sound trigger service to start opening the breath wake-up algorithm channel. Step 3, the sound trigger service notifies the sound trigger driver of the hardware abstraction layer to turn on the breath wake-up algorithm channel. Step 4, the breath wake-up algorithm is turned on and used to identify breath in order to be woken up.

[0090] In addition, the voice trigger service can also notify the microphone driver to start the microphone so that the microphone can receive the user's breath and voice at any time. In some embodiments, the voice trigger driver can also notify the microphone driver to start the microphone. Or the voice trigger driver includes the microphone driver, which directly drives the microphone to work, and this application is not limited to this.

[0091] The above steps 1 to 4 are the data flow process when the breath wake-up algorithm is turned on. This process can be when the user turns on the breath wake-up function through the settings in the application when using the electronic device for the first time. It can also be when voice-to-text conversion is required, the breath wake-up function is turned on through settings in the corresponding input method application.

[0092] The second part describes setting application priority. The process can be found in Figure 6 Follow steps 5 and 6 in the tutorial.

[0093] like Figure 6 As shown, taking the note application as an example, when the user opens the note application and clicks on the new note in the note interface (such as Figure 3b(as shown in i in the figure) to enter the input interface. At this time, the input method application obtains the click event and pulls up the input panel (input box) on the input interface. Step 5, the input method application pulls up the input panel (pop-up input panel) to realize the relevant input. IMMS can sense that the input panel is pulled up and obtain the identifier of the note application. Step 6, IMMS registers the wake-up message reception preparation based on the identifier of the note application. For example, the identifier of the note application is recorded in the registration list, and the identifier mark is recorded as 1 in the list corresponding to the record as the highest priority using the breath wake-up algorithm, and other non-highest priority application identifiers are recorded as 0 in the corresponding list to distinguish between the highest priority application and the non-highest priority application, so that after the user inputs the voice, the voice information can be sent to the input method application belonging to the note application.

[0094] The third part describes the process of voice input. This process can be referred to Figure 6 Steps 7 - 11 as shown.

[0095] like Figure 6 As shown, after the electronic device executes the above-mentioned breath wake-up algorithm and sets the application priority, if the user performs a second operation, such as picking up the phone and speaking close to the mouth, the electronic device executes step 7, continues to receive the user's voice, and obtains a voice stream. Step 8, the breath wake-up algorithm notifies the sound trigger driver to send event information related to the voice stream, such as session Id / buffer Size, to the sound trigger service. Step 9, the sound trigger driver passes the event information related to the voice stream to the sound trigger service (Sound Trigger), and the sound trigger service queries the current event receiving registration list to determine whether to send the breath wake-up event to the registrant with the highest priority (chat application). And execute step 10 to send the event to the input method application, which further converts the voice stream into text and displays it in the current chat application or note application.

[0096] Above Figure 6 In the process of implementing voice input described in the text, from the user's perspective, the user only needs to click on the input interface, pick up the phone and speak to realize the voice-to-text conversion of the application used. The operation is in line with normal operating habits, the action is smooth, and the user experience is improved.

[0097] refer to Figure 7 , Figure 7 A data flow diagram of a method for converting speech to text in a software framework of an electronic device according to another embodiment of the present application is shown. Figure 7As shown, the Android system can also be divided from top to bottom into an application layer, an application framework layer, an Android runtime (Android runtime) and a system library (not shown), as well as a hardware abstraction layer and a kernel layer.

[0098] like Figure 7 As shown, the application package may include many applications, such as a chat application, an input method application, and the like.

[0099] It should be noted that Figure 7 and Figure 6 The difference is that when the input method application in the application package transmits information, it transmits information through a public interface, the software development kit (SDK). The software development kit interface then transmits information to the application framework layer, or to each application in the application layer. In this way, the application framework layer can receive settings from any application in the application layer, and there is no need for the framework layer to adapt to each application, reducing the difficulty of third-party application adaptation.

[0100] like Figure 7 As shown, except for step 1, step 5 and step 10, the other steps are the same as Figure 6 The corresponding steps are the same.

[0101] For step 1, Figure 7 As shown, the user turns on the breath wake-up function based on the setting function of the input method application, and the setting module calls the SDK interface to send a breath wake-up function opening channel to the IMMS through the SDK interface.

[0102] For step 5, after the input method application pops up the input method panel based on the notification of the note application, the note application identifier and the notification of preparing for voice input are sent to the SDK interface, and then the SDK interface notifies the IMMS to prepare for voice input.

[0103] In some embodiments, the IMMS may directly sense the pop-up of the input panel and then prepare for voice input.

[0104] With respect to step 10, the sound trigger service module directly sends the corresponding event of the voice stream to the SDK interface, and the SDK interface notifies the corresponding application method application, and then the input method application converts the voice into text.

[0105] For steps 2 to 4 in the process of starting the breath wake-up algorithm, step 6 in the process of setting the application priority, and steps 7 to 9 in the process of voice input, the data flow of these steps is similar to Figure 6 The same is not repeated here.

[0106] In such Figure 7 In the process of communication between the application program and the application framework layer shown, communication is unified through the SDK interface. The SDK interface serves as an interface for communication between the application layer and the application framework layer, which facilitates all applications in the application layer to communicate with the framework layer to implement the speech-to-text method of the embodiment of the present application. From the perspective of technical development, the difficulty of third-party application adaptation is reduced.

[0107] The following uses a mobile phone as an example to describe the method for converting speech to text provided in the embodiment of the present application in detail. A plurality of applications are installed in the mobile phone. The method for converting speech to text provided in the embodiment of the present application can be applied to a mobile phone having the following features: Figure 4 The hardware structure shown in Figure 5-Figure 7 In an electronic device with the software structure shown in the figure, or with more or fewer components than shown in the figure, or with some components combined, or some components separated, or with different component arrangements, etc., and with similar hardware structures and software structures.

[0108] refer to Figure 8 , Figure 8 A flow chart of the method for converting speech to text in an embodiment of the present application is shown. Figure 8 As shown, the method can be executed by a mobile phone, including S810-S880.

[0109] S810, turn on the breath wake-up function.

[0110] In the embodiment of the present application, the operation timing corresponding to the breath awakening function can be turned on before the user inputs a voice. For example, the breath awakening function can be turned on when the mobile phone is used for the first time, or when the user wants to use voice-to-text conversion.

[0111] In an embodiment of the present application, the user can turn on the breath wake-up function through an automatically pop-up guide box when opening the application for the first time, as shown in the following figure: Figure 3a The interface operations and corresponding descriptions shown will not be described in detail here.

[0112] In some embodiments, the user can also find the interface for enabling the breath wake-up function through the settings in the input method application.

[0113] refer to Fig. 9 , Fig. 9 The operation interface diagram of the embodiment of the present application by setting up the breath awakening function is shown. Fig. 9The interface 910 shown in (a) of FIG. 1 can be obtained by going to Settings in the input method application - More settings - Input settings. In some embodiments, the input settings can also be found through the settings in the note application. The interface 910 includes a "breath wake-up voice to text" function. The user clicks on the corresponding icon to enter the following example: Fig.10 The breath awakening speech-to-text interface 920 (with Figure 3a The user can turn on or off the breath-awakened speech-to-text function by using the switch key 921 .

[0114] It should be noted that for the underlying data flow corresponding to the breath wake-up function, please refer to Figure 6 and Figure 7 The descriptions of steps 1 to 4 in enabling the breath wake-up function are not repeated here.

[0115] Combine the following Figure 3b , mobile phone interface operation diagram, and refer to Figure 8 Each step from S820 to S880 is described.

[0116] In S820, the note application is opened and a title interface corresponding to the note application is displayed. Figure 3b The interface 310 corresponds to (i) in FIG.

[0117] In some embodiments of the present application, a note application is used as an exemplary description of the first application. In some embodiments, the first application may also be a chat application, a notebook application, or the like. These applications have an input function and can display an input interface. Users can edit or input the content they want to express or interact with in the input interface. In the input interface, users can implement the interaction process through voice input, text input, voice-to-text input, etc.

[0118] S830, receiving an operation of clicking the title interface (corresponding to the first operation), displaying the input interface and popping up an input panel (input box) to which the input method application belongs.

[0119] This step can refer to Figure 3b (i) in the figure corresponds to the operation of the user clicking the new note function key. After the user clicks the new note function key 331, the note application calls the input method application to Figure 3b The input interface 340 shown in (ii) in FIG. 1 shows an input panel 341 (input box) to which the input method application belongs. At this time, the mobile phone enters a state where editing is possible at any time.

[0120] In S840 , the input method application is set as the highest priority for awakening by the user's breath.

[0121] It should be noted that the breath wake-up function can be applied to different applications, for example, it can also be used for voice assistants (third applications), etc. When the input method application is not prioritized, the voice assistant can use the breath wake-up function by default, so that the input method application cannot normally use the breath wake-up function. Therefore, in an embodiment of the present application, when the mobile phone determines that the user needs to perform voice input for the input method application, the input method application is set to the highest priority, so that the input method application can use the breath wake-up function. Among them, the highest priority recording method can be recorded in the form of an Excel table, that is, the input method application name is recorded in the table, and 1 is recorded in the corresponding position, and 1 is used to indicate the highest priority. Record 0 in the corresponding position of other applications, and 0 indicates that these applications cannot currently use the breath wake-up algorithm. In addition, it can also be recorded through a queue, and the input method application is placed at the top of the queue, and only the top application can use the breath wake-up algorithm. Through this setting method, when the user inputs a voice, the voice stream obtained by the breath wake-up function will be sent to the input method application based on the highest priority, and the input method completes the voice-to-text conversion, and finally the text is smoothly displayed in the input interface where the input method application is located.

[0122] In an embodiment of the present application, the mobile phone processor may write the identifier of the input method application, such as the name, unique ID, etc., in the registration table, and set the priority of the input method application to the highest. Alternatively, the priority identifier of the original default application is removed, and the input method application is marked with a priority identifier, such as the number 1. The implementation form of the priority setting in the embodiment of the present application is not limited.

[0123] In some embodiments, S840 may also include receiving a corresponding breath wake-up function callback event for the registered input method, so that the callback interface of the input method application continuously monitors the received voice stream, so that the input method application obtains the voice stream. The corresponding priority is the callback priority, that is, the callback interface of the input method application has the highest priority to obtain the voice stream.

[0124] In addition, the data flow of the underlying implementation of the above steps S830-S840 can be referred to Figure 6 and Figure 7 The description of steps 5 and 6 shown in the figure will not be repeated here.

[0125] S850, obtaining movement data of the mobile phone being lifted by the user and voice data input by the user. The interface operation of the mobile phone corresponding to this step can be referred to as Figure 3b The interface 340 shown in (ii) is as follows Figure 3bThe interface 350 shown in (iii) in FIG. 1 is a diagram showing an interface 350 in FIG. 1 . The corresponding action is that the user lifts up the mobile phone and speaks, which is consistent with the user's habit of using the mobile phone and makes the user feel more natural.

[0126] In the embodiment of the present application, the user lifts up the phone as the first action, and exhales as the second action. Such an operation is more in line with the user's habit of picking up the phone to speak.

[0127] It should be noted that in some embodiments, other actions may also be performed, for example, the mobile phone moves horizontally at an accelerated speed, or recognizes that the mobile phone moves at a constant speed but the rotation angle reaches a preset 90 degrees, etc. The present application does not limit the specific form of the action.

[0128] When the user wants to input voice, for example, the user picks up the phone and brings it close to the lips to a set distance, such as 5 cm, and inputs voice. The mobile phone can obtain these movement data through sensors, such as the acceleration sensor to obtain the mobile phone movement acceleration of 50m / s 2 , and the distance sensor detects that the distance to the object reaches 5cm. The microphone receives breath or voice signals, etc. After all these data are met, and when the user inputs voice, the mobile phone can directly obtain the user's voice stream. In an embodiment of the present application, the mobile phone can be based on a motion recognition model, which is obtained through continuous training, and can accurately identify that the user needs voice-to-text input based on the movement data and distance data obtained by the sensor. Among them, for the specific recognition method, reference can be made to the prior art, and this application will not be described in detail. The operation is simple, and it is natural and smooth for the user, and is easy to operate with one hand.

[0129] In the embodiments of the present application, the distance between the microphone of the mobile phone and the lips is set to be within 5 cm. In some embodiments, it can also be within 8 cm or 10 cm, which is not limited in the present application.

[0130] S860: Acquire a voice stream continuously input by the user.

[0131] In an embodiment of the present application, the mobile phone obtains the voice input by the user through the microphone, and displays the audio amplitude of the voice stream being input in the interface of the mobile phone so that the user can understand that the current voice input is valid.

[0132] like Figure 3b As shown in (iii), when the user lifts the phone close to his mouth to speak, the breath wake-up function uses the breath wake-up algorithm to obtain the corresponding voice stream, and displays it through audio amplitude in the interface 350 of the mobile phone, so that the user can more intuitively observe the voice input.

[0133] For the underlying data stream of obtaining and transmitting the voice stream to the input method application, please refer to Figure 6 and Figure 7 The descriptions of steps 7 to 10 in the previous section are not repeated here.

[0134] S870, convert the voice stream into text and display it in the input interface corresponding to the note application.

[0135] like Fig.11 As shown in (d) in the figure, in the interface 1140 , the corresponding voice stream is converted into text “How is the weather today?” The user can directly obtain the text information in the interface 1140 .

[0136] For the user, the above operation process is just to click the edit box and hold the phone close to the mouth to speak during voice input, and then voice to text can be converted. The entire operation process does not require the user to long-term control, or to click the voice input function key when inputting voice. The operation is simple, natural, and smooth, which provides convenience for one-handed operation and improves the user experience.

[0137] In some embodiments of the present application, the mobile phone can execute the above Figure 8 You can complete the voice-to-text input by the operation method shown in the figure. At the same time, you can also choose to use the existing conventional operation method. For example, long press Figure 3b In actual use, the user can use the voice-to-text method of the embodiment of the present application or the existing voice input key to complete the voice input, so that the user can flexibly choose according to the needs and have a better experience.

[0138] In some embodiments of the present application, Figure 3b The interface 350 of (iii) and Figure 3b The interface 360 ​​shown in (iv) of FIG. 3 is shown. When the preset time period has elapsed after the end of the current voice stream, for example, there is no voice input for 3 seconds (third operation), the mobile phone can determine that the voice input has ended, and the position of the audio amplitude 351 corresponding to the voice is restored to the original letter or symbol interface 361 of the input panel. In this way, the user can input the next voice at any time, or select handwriting input, which is convenient for the user to switch the input mode.

[0139] In some embodiments, it can also be instant voice-to-text conversion, that is, when the user inputs voice, the mobile phone directly converts the voice into text without requiring confirmation of the end of the voice stream.

[0140] In some embodiments, when the user clicks the function key (third operation) to end the speech in the interface of the input method application. Figure 3bThe control 11 shown in (iii) in the figure, at this time, the control 11 displays "finished speaking" to prompt the user that the voice input is finished. The user can click the control 11 to indicate that the current voice input is finished.

[0141] In some embodiments, when the user has not finished the voice input, for example, the voice has not reached the preset duration of 3 seconds, the mobile phone interface can maintain the state of the audio input with musical notes. Fig.10 , Fig.10 The following is a schematic diagram of the interface for voice input in an embodiment of the present application. Fig.10 As shown in (a), the interface when the user is inputting voice is 1010, and the amplitude of the audio in the interface 1010 is relatively large. Fig.10 In the interface 1020 shown in (b), the amplitude of the audio in the interface 1020 is small or uniform.

[0142] In the above embodiment, the description is made by taking the user using the note application as an example. In some embodiments, when the user uses other applications, such as a chat application, the conditions for triggering the display of the input box may be different.

[0143] refer to Fig.11 As shown, Fig.11 The following is a schematic diagram showing the operation interface corresponding to opening a chat application in a mobile phone of an embodiment of the present application. Fig.11 As shown, after the user opens the chat application, the input interface 1110 is displayed, and the input box is not displayed or popped up. When the user clicks the edit box 1111, the user enters the Fig.11 In the interface 1120 shown in (b) of FIG. 1120, an input panel 1121 (input box) pops up or is displayed in the interface 1120. That is, the input panel can be opened together with the input interface, or it can be opened after the input interface is displayed, and this application does not limit this. After that, the user lifts the mobile phone and brings it close to his mouth to speak, and the mobile phone displays the following Fig.11 In the interface 1130 shown in (c), the audio amplitude is displayed in the input panel 1131, and the text after speech conversion is displayed in the information input field 1132, such as "How is the weather today?".

[0144] The following is an explanation of the interaction process of each module of the method for converting speech to text in conjunction with a mobile phone.

[0145] refer to Fig.12 , Fig.12 The flowchart of the interaction between the modules of the method for converting speech to text in the mobile phone of the embodiment of the present application is shown. Figure 4 The hardware structure shown may also include Figure 5-Figure 7 The software architecture is shown in Figure 1. Fig.12As shown, the mobile phone includes software modules such as input method apk, input method service, sound trigger service, sound trigger driver and chip algorithm. Among them, the input method apk corresponds to Figure 5 The input method application, input method service and sound trigger service in the application layer are located in the application Figure 5 The application framework layer shown in the figure, the sound trigger driver is located in Figure 5 The hardware abstraction layer shown, the chip algorithm corresponds to Figure 5 The breath awakening algorithm in the kernel layer.

[0146] like Fig.12 As shown in the figure, based on the user's operation, each module is divided into four parts based on the data processing process of the operation. Fig.13 The four parts are shown as ①, ②, ③ and ④.

[0147] Among them, ① is the process of starting the breath awakening algorithm, including S01-S04. The data flow of these steps can be referred to Figure 6 or Figure 7 Description of steps 1, 2, 3, and 4 in .

[0148] S01, the input method apk receives the user's operation of turning on the breath awakening algorithm switch and creates an event callback.

[0149] Among them, the input method apk, as an independent module, can obtain information from the settings, such as obtaining the switch to turn on the breath wake-up. Based on the opening operation, the input method apk creates an event callback interface to obtain the voice stream of the voice.

[0150] The input method apk notifies the input method service of the event of turning on the breath wake-up switch.

[0151] S02, the input method service notifies the sound trigger service to use the breath wake-up algorithm pathway and stores the event callback.

[0152] Among them, after the event callback is stored, after the input method service obtains the input panel displayed in the interface, it will notify the sound trigger service and set the highest priority for the event callback so that the input method apk can be ready to obtain the voice stream at any time.

[0153] S03, the sound trigger service notifies the sound trigger driver to use the breath wake-up algorithm pathway.

[0154] For example, the sound trigger service sends a signal to open the breath awakening algorithm channel to the sound trigger driver, so that the sound trigger driver notifies the next level to use the breath awakening algorithm channel.

[0155] S04, the sound triggers the driver to notify the chip algorithm to turn on the breath awakening algorithm.

[0156] In an embodiment of the present application, when the breath wake-up algorithm is turned on, if the user does not turn off the algorithm switch and the mobile phone is in normal startup, the breath wake-up algorithm can remain turned on, and the user can use the breath wake-up algorithm at any time using the mobile phone.

[0157] Among them, S01-S04 and Figure 8 Corresponding to S810 in, for specific explanation, please refer to the description in S810.

[0158] ② is the process of setting application priority, including S05-S07. The data flow of these steps can be referred to Figure 6 or Figure 7 The description of steps 5 and 6 in .

[0159] S05: The input method service detects that an input panel is displayed in the interface (first operation).

[0160] In some embodiments of the present application, the user may click on the edit bar of the input interface to pop up an input panel, triggering the input method service to execute S06.

[0161] S06: The input method service sends an event callback to the sound trigger service and sets the highest priority.

[0162] Among them, setting the highest priority is used to indicate that the current highest priority user is the input method APK, and when there is voice input, the voice message is sent to the input method APK first.

[0163] S07, the sound triggers the service to register an event callback and updates the priority in the list.

[0164] In some embodiments, the highest priority can be determined in the form of a queue, for example, the callback (input method APK) with the highest priority is placed at the top of the queue. Each time a voice stream is obtained, it is sent to the callback at the top of the list first. In some embodiments, the identifier of the callback can also be marked, for example, the user marked as 1 is given priority to use the chip algorithm, etc. This application does not limit the way to implement the priority.

[0165] The above S05-S07 and Figure 8 The steps S830 and S840 correspond to each other, and for detailed explanations, please refer to the descriptions in S830 and S840.

[0166] ③ is the process of voice input, including S08-S12. The data flow of these steps can be referred to Figure 6 or Figure 7 The description of steps 7 and 11 in .

[0167] S08: The chip algorithm recognizes that the user lifts up the phone and speaks.

[0168] In an embodiment of the present application, the chip algorithm can integrate two modules, namely, an action recognition module and a breath recognition module. When the user lifts the mobile phone, the acceleration sensor of the mobile phone can obtain the data of the acceleration of the mobile phone and send it to the action recognition module, and the action recognition module recognizes the user's lifting action. After the action is recognized, the user speaks. At this time, the microphone obtains the user's voice or breath and converts it into an electrical signal corresponding to the breath. The electrical signal is forwarded to the breath recognition module, and the breath recognition module recognizes that there is breath input based on the electrical signal. After recognizing the lifting and speaking, the chip algorithm is triggered to continuously obtain the voice stream.

[0169] In the embodiment of the present application, lifting the mobile phone and speaking is used as an example of the second operation. In some embodiments, other operations are also possible. For example, the user taps the mobile phone twice and then speaks close to the mouth, or shakes the mobile phone three times and speaks close to the mouth, etc.

[0170] S09, the chip algorithm reports the breath awakening event to the sound trigger driver.

[0171] S10, the sound trigger driver reports the breath awakening event to the sound trigger service and specifies the session address, selects the highest priority registrant and triggers the callback. For example, the highest priority registrant is the input method apk, and the callback interface of the input method apk is triggered.

[0172] S11, the sound trigger service triggers the input method APK callback and passes parameters and voice stream to trigger the input method APK callback to continuously obtain the voice stream.

[0173] S12, triggering the input method APK to convert the voice corresponding to the voice stream into text. Thereafter, the processor controls the display screen to display the text content.

[0174] The above S08-S12 and Figure 8 The S850 and S870 in the figure correspond to each other. For specific explanations, please refer to the descriptions in S850 and S870.

[0175] ④ The process of closing voice input for the user, including S13-S15.

[0176] S13, the input method service detects that the input method panel is retracted (the fourth operation).

[0177] S14, the input method service notifies the sound trigger service to cancel the callback registration.

[0178] S15, the sound trigger service updates the registration list and restores the priority of the original application in the list, that is, cancels the priority of the application corresponding to the input method APK and restores the priority in the original registration list.

[0179] ⑤ The process of the user raising the phone to speak again. This includes: Fig.12 S16 in.

[0180] S16: The chip algorithm detects again that the user lifts up the phone and speaks.

[0181] For example, after restoring the priority in the registration list, the highest priority of the YOYO application is restored in the list. At this time, the chip algorithm forwards the obtained voice stream to YOYO. This enables seamless switching between different applications according to user needs, without the need for additional user operations, and greatly improves the experience.

[0182] In some embodiments of the present application, when the user closes the input panel so that the input panel is not in the interface of the currently running input method application (the third operation), it indicates that the user has ended the voice-to-text input of the current input method application. At this time, the processor of the mobile phone can cancel the highest priority of the breath wake-up function corresponding to the input method application. For example, the highest priority callback priority corresponding to the input method application is deleted from the registration list, the original callback priority arrangement in the registration list is restored, and when the voice stream is received, it is no longer sent to the input method application in priority.

[0183] In addition, it indicates that the user has ended the voice-to-text input operation of the current input method application. It can also be that the user closes the input method application, or the input method application enters the background. At this time, the mobile phone will cancel the priority of the input method application and restore the priority of the system's original default application.

[0184] According to the method of voice-to-text conversion in the embodiment of the present application, in the process of triggering voice-to-text conversion, the user does not need to press the voice input function key for a long time, nor does he need to click the interface button or other subtle operations to realize voice input. The whole operation is natural and smooth, and it is convenient for one-handed operation, which can greatly improve the user experience.

[0185] The present application also provides an electronic device, including:

[0186] a memory for storing instructions to be executed by one or more processors of the device, and

[0187] A processor is used to execute the above embodiment in combination with Figures 5 to 12 The method explained.

[0188] The present application also provides a computer-readable storage medium storing a computer program. When the computer program is executed by a processor, the processor executes the above-mentioned embodiment. Figures 5 to 12 The method explained.

[0189] The present application also provides a computer program product comprising instructions, which, when executed on an electronic device, enables a processor to execute the above-mentioned embodiments. Figures 5 to 12 The method explained.

[0190] Reference now Fig.13 , which is a block diagram of a SoC (System on Chip) 1300 according to an embodiment of the present application. Fig.13 In the FIG, similar components have the same reference numerals. In addition, the dashed boxes are optional features of more advanced SoCs. Fig.13 In the embodiment, SoC 1300 includes: an interconnect unit 1350, which is coupled to an application processor 1310; a system agent unit 1380; a bus controller unit 1390; an integrated memory controller unit 1340; a group or one or more coprocessors 1320, which may include integrated graphics logic, an image processor, an audio processor, and a video processor; a static random access memory (SRAM) unit 1330; and a direct memory access (DMA) unit 1360. In one embodiment, the coprocessor 1320 includes a dedicated processor, such as, for example, a network or communication processor, a compression engine, a general purpose computing on GPU (GPGPU), a high throughput integrated multi-core architecture (Many Integrated Core, MIC) processor, or an embedded processor.

[0191] The static random access memory (SRAM) unit 1330 may include one or more computer-readable media for storing data and / or instructions. The computer-readable storage medium may store instructions, specifically, temporary and permanent copies of the instructions. The instructions may include: when executed by at least one unit in the processor, the Soc 1500 executes the method according to the above embodiment, which may be specifically referred to in the above embodiment. Figures 5 to 12 The methods explained will not be repeated here.

[0192] The various embodiments of the mechanism disclosed in the present application can be implemented in hardware, software, firmware or a combination of these implementation methods. The embodiments of the present application can be implemented as a computer program or program code executed on a programmable system, which includes at least one processor, a storage system (including volatile and non-volatile memory and / or storage elements), at least one input device and at least one output device.

[0193] Program code can be applied to input instructions to perform the functions described in this application and generate output information. The output information can be applied to one or more output devices in a known manner. For the purposes of this application, a processing system includes any system having a processor such as, for example, a digital signal processor (DSP), a microcontroller, an application specific integrated circuit (ASIC), or a microprocessor.

[0194] Program code can be implemented with high-level programming language or object-oriented programming language to communicate with the processing system. When necessary, program code can also be implemented with assembly language or machine language. In fact, the mechanism described in this application is not limited to the scope of any specific programming language. In either case, the language can be a compiled language or an interpreted language.

[0195] In some cases, the disclosed embodiments may be implemented in hardware, firmware, software, or any combination thereof. The disclosed embodiments may also be implemented as instructions carried or stored on one or more temporary or non-temporary machine-readable (e.g., computer-readable) storage media, which may be read and executed by one or more processors. For example, instructions may be distributed over a network or by other computer-readable media. Therefore, machine-readable media may include any mechanism for storing or transmitting information in a machine (e.g., computer) readable form, including, but not limited to, floppy disks, optical disks, optical disks, compact disc read-only memories (Compact Disc Read Only Memory, CD-ROMs), magneto-optical disks, read-only memories (Read Only Memory, ROM), random access memories (RAM), erasable programmable read-only memories (Erasable Programmable Read Only Memory, EPROM), electrically erasable programmable read-only memories (Electrically Erasable Programmable Read Only Memory, EEPROM), magnetic cards or optical cards, flash memory, or a tangible machine-readable memory for transmitting information (e.g., carrier waves, infrared signal digital signals, etc.) using the Internet in electrical, optical, acoustic, or other forms of propagation signals. Accordingly, machine-readable media include any type of machine-readable media suitable for storing or transmitting electronic instructions or information in a form readable by a machine (eg, a computer).

[0196] In the accompanying drawings, some structural or method features may be shown in a specific arrangement and / or order. However, it should be understood that such a specific arrangement and / or order may not be required. Instead, in some embodiments, these features may be arranged in a manner and / or order different from that shown in the accompanying drawings of the specification. In addition, the inclusion of structural or method features in a particular figure does not mean that such features are required in all embodiments, and in some embodiments, these features may not be included or may be combined with other features.

[0197] It should be noted that the units / modules mentioned in the various device embodiments of the present application are all logical units / modules. Physically, a logical unit / module can be a physical unit / module, or a part of a physical unit / module, or can be implemented as a combination of multiple physical units / modules. The physical implementation method of these logical units / modules themselves is not the most important. The combination of functions implemented by these logical units / modules is the key to solving the technical problems proposed by the present application. In addition, in order to highlight the innovative part of the present application, the above-mentioned device embodiments of the present application do not introduce units / modules that are not closely related to solving the technical problems proposed by the present application, which does not mean that there are no other units / modules in the above-mentioned device embodiments.

[0198] It should be noted that in the examples and description of this patent, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "including one" do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0199] Although the present application has been illustrated and described with reference to certain preferred embodiments thereof, it will be apparent to those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the present application.

Claims

1. A method for converting speech to text, applied to electronic equipment, characterized in that: The method comprises: In response to the first operation, an input box is displayed in a first interface of the first application, wherein the breath wake-up function of the electronic device is turned on; In response to the display of the input box, setting the second application to which the input box belongs as the highest priority for using the breath wake-up function; wherein the breath wake-up function triggers the second application with the highest priority to continuously receive voice data when the user moves the electronic device and receives the user's breath; In response to the user moving the electronic device and detecting the user's breath, the second application of the electronic device continues to receive voice data and converts the voice data into text; The electronic device displays the converted text in the first interface.

2. The method according to claim 1, characterized in that The first application is a note application, and the first operation is an operation of creating a new note or an editing operation on a note details page.

3. The method according to claim 1, characterized in that The first application is an instant messaging application, and the first operation is an operation performed in a dialog interface of the instant messaging application.

4. The method according to any one of claims 1 to 3, characterized in that: In response to the user moving the electronic device and detecting the user's breath, the second application of the electronic device continues to receive voice data and converts the voice data into text, including: It is detected that the movement acceleration of the electronic device reaches a preset value, and the user's breath is detected through the breath wake-up function, and the second application of the electronic device continues to receive voice data and converts the voice data into text.

5. The method according to any one of claims 1 to 4, characterized in that: Setting the second application to which the input box belongs as the highest priority for using the breath wake-up function includes: Register the callback event of the second application to the breath awakening function in the registration list, and set the callback priority corresponding to the second application to the highest priority.

6. The method according to claim 4, characterized in that The method further comprises: In response to the third operation, the electronic device cancels the highest priority of the second application using the breath wake-up function, and no longer sends the voice data to the second application when it is received.

7. The method according to claim 6, characterized in that The third operation comprises: An operation of closing the input box; or The operation of removing the running interface of the second application from the foreground.

8. The method according to claim 6, characterized in that Also includes: After canceling the highest priority of the second application using the breath wake-up function, the highest priority is restored to the third application; In response to the user moving the electronic device and detecting the user's breath, the third application of the electronic device continues to receive voice data.

9. The method according to claim 8, characterized in that The third application is a voice assistant application.

10. An electronic device, characterized in that: include: a memory for storing instructions to be executed by one or more processors of the device, and A processor is used to execute the instructions so that the electronic device executes the method according to any one of claims 1 to 9.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the processor executes the method according to any one of claims 1 to 9.

12. A chip structure, characterized in that: The method comprises at least one chip, wherein the at least one chip is used to execute the method according to any one of claims 1 to 9.

13. A computer program product comprising instructions, characterized in that When the computer program product runs on an electronic device, it enables the processor to execute any one of the methods described in 1-9.

Images