Method for invoking speech recognition engine and electronic device

Abstract

This application provides a method for invoking a speech recognition engine and an electronic device, relating to the field of electronic technology, for reducing latency during ASR engine switching and lowering unnecessary power consumption. The method is applied to an electronic device having a first voice control function; the electronic device includes a speech recognition ASR engine and a voice kit; a first voice control module corresponding to the first voice control function interfaces with the ASR engine through the voice kit; the method includes: the first voice control module receiving a first voice control trigger event; when the voice kit is in a running state, the first voice control module responds to the first voice control trigger event by sending a first invocation request through the voice kit's initialization interface for the ASR engine; the ASR engine responds to the first invocation request by initializing its service data to the first service data corresponding to the first voice control module, thereby providing services for the first voice control function.

Description

Technical Field

[0001] This application relates to the field of electronic technology, and in particular to a method for invoking a speech recognition engine and an electronic device. Background Technology

[0002] With technological advancements, it is now possible for users to control electronic devices via voice. A single electronic device may have two or more different voice control functions. These different voice control functions can be used in different scenarios to achieve different voice control effects. Voice control functions typically require the use of an Automatic Speech Recognition (ASR) engine to recognize the captured user's voice.

[0003] Different voice control functions can share the same ASR engine, and some voice control functions all need to interface with and call the ASR engine through the same voice kit. In related technologies, the ASR engine is a single instance in electronic devices, allowing only one service to access it at a time; that is, the ASR engine can only provide services for one voice control function at a time. Currently, in scenarios where electronic devices switch between different voice control functions, there is a problem of long latency during service switching, which can easily lead to unnecessary power consumption waste. Summary of the Invention

[0004] This application provides a method for invoking a speech recognition engine and an electronic device to reduce latency during ASR engine switching and reduce unnecessary power consumption.

[0005] To achieve the above objectives, the embodiments of this application adopt the following technical solutions:

[0006] Firstly, a method for invoking a speech recognition engine is provided. This method is applied to an electronic device that has a first voice control function. The electronic device includes a speech recognition engine (ASR) and a voice kit (voicekit). A first voice control module corresponding to the first voice control function interfaces with the ASR engine through the voicekit. The method includes:

[0007] The first voice control module receives a first voice control trigger event. With VoiceKit running, the first voice control module responds to the first voice control trigger event by sending a first call request through VoiceKit's initialization interface for the ASR engine. This avoids the step of unregistering and re-initializing VoiceKit when the first voice control module calls the ASR engine. Instead, the ASR engine responds to the first call request by initializing its service data to the first service data corresponding to the first voice control module, enabling the ASR engine to provide services for the first voice control function.

[0008] In this solution, when the voice control function needs to call the ASR engine through VoiceKit, the status of VoiceKit is first checked. If VoiceKit is already running, the first voice control module does not need to unregister and initialize VoiceKit. Instead, it directly calls the ASR engine's initialization interface through VoiceKit to initialize VoiceKit and provide services for the first voice control function. This avoids repeated unregistration and initialization of VoiceKit in scenarios where two or more voice control functions using the same ASR engine are launched sequentially, thus preventing repeated unbinding and binding between other modules within VoiceKit and other engines. This reduces latency during ASR engine service switching and minimizes unnecessary power consumption.

[0009] In one possible implementation of the first aspect, before the first voice control module sends a first call request through the voicekit's initialization interface for the ASR engine in response to a first voice control trigger event when the voicekit is in the started state, the first voice control module can first determine whether the current user of the ASR engine is the first voice control module. If the current user of the ASR engine is not the first voice control module, then when the voicekit is in the started state, the first voice control module responds to the first voice control trigger event and sends a first call request through the voicekit's initialization interface for the ASR engine.

[0010] In this scheme, if the current user of the ASR engine is not the first voice control module itself, it means that the ASR engine cannot be directly used by the first voice control module and needs to be reinitialized. Therefore, only after determining that the current user of the ASR engine is not the first voice control module will the first voice control module send an initialization interface call request to the ASR engine through voicekit to initialize the ASR engine.

[0011] In one possible implementation of the first aspect, after the ASR engine initially completes its response to the voice control module's call request, it generates the call parameters corresponding to the current user and saves these parameters as the current call parameters. Furthermore, the ASR engine can also return these current call parameters to the current user for storage. However, if the current user of the ASR engine changes, the ASR engine will update the current call parameters accordingly.

[0012] Therefore, if the first voice control module has called the ASR engine and has not actively exited the call to the ASR engine, the first voice control module will save the call parameters from the ASR engine. These call parameters correspond to the first voice control module. When the first voice control trigger event is received, if it is detected that the first voice control module has saved the ASR engine's call parameters, the first voice control module can first determine whether the current call parameters of the ASR engine are consistent with the call parameters it has saved, in order to determine whether the current call parameters of the ASR engine belong to the first voice control module.

[0013] In one possible implementation of the first aspect, if the first voice control module stores the calling parameters of the ASR engine, but the calling parameters are inconsistent with the current calling parameters of the ASR engine, it can be determined that the current user of the ASR engine is not the first voice control module.

[0014] Furthermore, if the first voice control module stores the ASR engine's calling parameters, it indicates that the first voice control module has called the ASR engine. If the calling parameters of the first voice control module are inconsistent with the current calling parameters of the ASR engine, it means that the ASR engine was preempted by other voice control functions during the first voice control module's use of the ASR engine. In this case, the first voice control module can resume use of the ASR engine after the current user of the ASR engine exits. Therefore, the first voice control triggering event includes: a recording status change event triggered when the current user of the ASR engine exits.

[0015] In another possible implementation of the first aspect, if the first voice control module does not store the calling parameters of the ASR engine, it indicates that the first voice control module has not called the ASR engine, and therefore the current user of the ASR engine is not the first voice control module. Furthermore, the fact that the first voice control module has not called the ASR engine means that the first voice control trigger event can specifically be a startup event for the first voice control function.

[0016] In one possible implementation of the first aspect, if the current user of the ASR engine is the first voice control module, then the current ASR engine can be used by the first voice control module. In this case, the first voice control module does not need to reinitialize the ASR engine, but can directly use the ASR engine for voice control through VoiceKit.

[0017] In one possible implementation of the first aspect, if the first voice control module stores the call parameters from the ASR engine and the call parameters are consistent with the current call parameters of the ASR engine, it indicates that the current user of the ASR engine is the first voice control module.

[0018] In this scheme, before initiating a call request to the ASR engine in response to a first voice control trigger event, the first voice control module first determines whether the current user of the ASR engine is the first voice control module. This avoids the first voice control module repeatedly initializing the ASR engine via VoiceKit when the current user is indeed the first voice control module, thus reducing unnecessary power consumption.

[0019] In one possible implementation of the first aspect, if the first voice control module does not store the call parameters from the ASR engine and a startup event for the first voice control module is detected, the first voice control module needs to call the ASR engine. The ASR engine then needs to switch from the current user to the first voice control module. Therefore, in response to the first call request, the ASR engine initializes its service data to the first service data corresponding to the first voice control module. Specifically, this may include: the ASR engine, in response to the first call request, first deregistering the service data corresponding to the current user. Then, the ASR engine initializes its service data to the first service data corresponding to the first voice control module. In this way, the ASR engine directly performs deregistration and re-initialization according to the call request, without requiring voicekit deregistration and initialization. This avoids the latency caused by unnecessary deregistration and initialization of the voicekit when switching services, reduces power consumption waste, and improves the efficiency of service switching.

[0020] In one possible implementation of the first aspect, if the calling parameters stored in the first voice control module are inconsistent with the current calling parameters of the ASR engine, the current user of the ASR engine can be the second voice control module corresponding to the second voice control function of the electronic device. Accordingly, the current calling parameters of the ASR engine are the second calling parameters corresponding to the second voice control function.

[0021] In this scheme, before the first voice control module receives the first voice control trigger event, the method further includes: the second voice control module receiving a second start event for the second voice control function; the second voice control module responding to the second start event sending a second call request through the initialization interface of the ASR engine via VoiceKit; responding to the second call request, initializing the service data of the ASR engine to the second service data corresponding to the second voice control function; the second voice control module receiving an exit event for the second voice control function; the second voice control module responding to the exit event triggering a recording status change event and sending a third call request through the deregistration interface of the ASR engine via VoiceKit; and the ASR engine responding to the third call request deregistering the second service data. Saving the first call parameter as the current call parameter of the ASR engine includes: the ASR engine updating the current call parameter from the second call parameter to the first call parameter.

[0022] In this scheme, before the first voice control module receives the first voice control trigger event, it first calls the ASR engine. Then, the second voice control module starts and calls the ASR engine, and then exits using the ASR engine. Afterwards, the first voice control module can re-request to call the ASR engine after detecting that the second voice control module has exited using the ASR engine, thus restoring the first voice control function. The first call request initiated by the first voice control module is initiated after the second voice control module exits using the ASR engine. For the ASR engine, it needs to first deregister the second service data corresponding to the second voice control module according to the third call request initiated by the second voice control module for the deregistration interface, and then respond to the first call request by initializing the ASR engine's service data to the first service data corresponding to the first voice control module. This achieves the effect of restoring the first voice control function.

[0023] In this way, during the process of the second voice control module exiting the use of the ASR engine and the first voice control module resuming its use of the ASR engine, the first voice control module can also directly send an initialization call request to the ASR engine through the voicekit while the voicekit is running. This reduces the steps of voicekit deregistration and initialization during the ASR engine's service switching process, reducing latency and power consumption waste.

[0024] In one possible implementation of the first aspect, when the current user of the ASR engine is not the first voice control module, the first voice control module sends an initialization request (first call request) to the ASR engine via VoiceKit. Upon receiving the first call request, the ASR engine can determine whether the current situation meets preset switching conditions. If so, it allows the current user to switch to the first voice control module. The preset switching conditions indicate that the ASR engine allows the current user (not the first voice control module) to switch to the first voice control module. This ensures the normal operation of the ASR engine for the current user.

[0025] In one possible implementation of the first aspect, if VoiceKit is in an inactive state when the first voice control module receives the first voice control trigger event, then the first voice control module, in response to the first voice control trigger event, first sends a first call request to VoiceKit. VoiceKit starts in response to the first call request. Then, VoiceKit sends the first call request to the ASR engine; the ASR engine, in response to the first call request, initializes its service data to the first service data corresponding to the first voice control module.

[0026] In a second aspect, an electronic device is provided, comprising: a processor, a memory, and a microphone; the memory and the microphone are respectively coupled to the processor. The memory is used to store computer execution instructions, and when the electronic device is running, the processor executes the computer execution instructions stored in the memory to cause the electronic device to perform a speech recognition engine invocation method as described in any of the first aspects above.

[0027] Thirdly, a computer-readable storage medium is provided that stores instructions which, when executed on a computer, enable the computer to execute the invocation method of the speech recognition engine of any of the first aspects described above.

[0028] Fourthly, a computer program product containing instructions is provided, which, when run on an electronic device, enables the electronic device to execute the invocation method of the speech recognition engine of any of the first aspects described above.

[0029] Fifthly, an apparatus (e.g., a system-on-a-chip) is provided, comprising a processor for supporting an electronic device in performing the functions described in the first aspect above. In one possible design, the apparatus further comprises a memory for storing program instructions and data necessary for the electronic device. When the apparatus is a system-on-a-chip, it may be composed of chips or may include chips and other discrete devices.

[0030] The technical effects of any of the design methods in aspects two through five can be found in the technical effects of different design methods in aspect one, and will not be repeated here. Attached Figure Description

[0031] Figure 1 A schematic diagram illustrating a method for controlling a terminal device using voice according to an embodiment of this application;

[0032] Figure 2 A schematic diagram illustrating a method for controlling a terminal device using voice according to an embodiment of this application;

[0033] Figure 3 A schematic diagram illustrating a scenario for setting a preset switch for a voice control function, provided in an embodiment of this application;

[0034] Figure 4 A schematic diagram of a voice-controlled mobile phone interface for performing operations, provided as an embodiment of this application;

[0035] Figure 5 A schematic diagram of a voice-controlled mobile phone interface for performing operations, provided as an embodiment of this application;

[0036] Figure 6 A flowchart illustrating a method for invoking a speech recognition engine in a related art, provided as an embodiment of this application;

[0037] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of this application;

[0038] Figure 8 A software architecture diagram of an electronic device provided in an embodiment of this application;

[0039] Figure 9 This is a schematic diagram illustrating the interaction of various modules when using voice control on a mobile phone, as provided in an embodiment of this application.

[0040] Figure 10 A flowchart illustrating a method for invoking a speech recognition engine, provided in an embodiment of this application;

[0041] Figure 11 A flowchart illustrating a method for invoking a speech recognition engine, provided in an embodiment of this application;

[0042] Figure 12 A flowchart illustrating a method for invoking a speech recognition engine, provided in an embodiment of this application;

[0043] Figure 13 A flowchart illustrating a method for invoking a speech recognition engine, provided in an embodiment of this application;

[0044] Figure 14This is a schematic diagram of a chip system provided in an embodiment of this application. Detailed Implementation

[0045] The following is a brief introduction to the technical terms that may be involved in the embodiments of this application.

[0046] Voice control function:

[0047] Many electronic devices support voice control. These devices use microphones to capture user voice recordings, analyze and recognize the voice, and execute the corresponding commands, allowing users to control the devices via voice.

[0048] Generally, to conserve power and prevent accidental triggering of electronic devices, voice control functions need to be enabled before use. For example, a user inputs a preset word (called a wake-up word) into the electronic device to wake it up. Once awakened, the device can execute the corresponding voice command, thus activating the voice control function. For instance, a user can enable or disable voice control by toggling preset switches on or off the device's user interface.

[0049] Voice control functions may have different names in different electronic devices, such as "voice control," "intelligent voice," "voice assistant," "see and speak," "voice command," "command at will," and "intelligent AI." The specific implementation of voice control functions with different names may also differ.

[0050] The following examples illustrate several different implementations of voice control functionality.

[0051] Voice assistant:

[0052] Before a user can control an electronic device using a voice assistant, the device needs to be woken up. In one example, the device is woken up when a wake word is detected by the user's voice input. In another example, the device is woken up when the user presses and holds the power button. In yet another example, the device is woken up when the user's breath is detected while inputting voice commands.

[0053] Typically, before an electronic device is activated, its microphone operates in a power-saving mode (e.g., searching for signals at low power) to pick up ambient sounds. The voice captured by the microphone is only used for wake word detection at the kernel level; the corresponding recording channel for the voice assistant is not activated in the electronic device's system or drivers.

[0054] In response to user actions (such as receiving a wake-up word from the user's voice), the electronic device is awakened, and the corresponding recording channel of the voice assistant is activated in the system and driver. After the electronic device is awakened, the voice (audio stream) captured by the microphone is sent to the voice assistant application for processing through the corresponding recording channel of the voice assistant. In this way, the electronic device can execute the user's voice commands, enabling the user to control the electronic device through voice; it can also realize functions such as dialogue with the user.

[0055] For example, Figure 1 This illustration shows a scenario where a user controls their phone using a voice assistant. Figure 1 As shown, the phone displays its home screen, and the user inputs the voice command "Hello YOYO". In response to receiving the wake-up word "Hello YOYO", the phone is woken up. For example, after being woken up, the phone plays the voice command "I'm here" to notify the user that the phone has been activated. After the phone is woken up, the user can control it via voice. For example, as... Figure 1 As shown, the user inputs the voice command "Open Calendar" into the phone. The phone parses and recognizes the user's voice and executes the command corresponding to "Open Calendar." For example, in response to receiving the user's voice command "Open Calendar," the phone launches the Calendar application.

[0056] For example, Figure 2 This illustration shows another scenario where a user controls their phone using a voice assistant. (Example:) Figure 2 As shown, the phone displays the home screen, and the user presses and holds the power button. In response to the user's press and hold of the power button, the phone is woken up. Once woken up, the user can control the phone via voice. For example, as... Figure 2 As shown, the user inputs the voice command "Open Calendar" into the phone. The phone parses and recognizes the user's voice and executes the command corresponding to "Open Calendar." For example, in response to receiving the user's voice command "Open Calendar," the phone launches the Calendar application.

[0057] Understandably, in other instances, the phone can also respond to other user actions on the power button (such as double-tapping) to activate the voice assistant.

[0058] Alternatively, in some other embodiments, the phone may also display an icon corresponding to the voice assistant on the interface. Upon detecting an action on the icon corresponding to the voice assistant, the phone can activate the voice assistant.

[0059] In some implementations, after the electronic device is woken up, the user can issue a command to the device by inputting voice, and the device will execute the command. Once the electronic device has executed a command, or if it does not receive a command from the user via voice within a certain period after being woken up (e.g., within 8 seconds), the device will no longer respond to user voice commands. For example, the electronic device may close the recording channel corresponding to the voice assistant. The user needs to input a wake-up word again to wake up the device before they can issue commands via voice again. In other words, after being woken up, the electronic device enters a "short-reception" state, and can respond to user voice commands for a short period (e.g., within 8 seconds).

[0060] In some implementations, the electronic device, while connected to a network, supports continuous dialogue upon being woken up, allowing users to engage in ongoing conversations. After each broadcast, the electronic device resumes audio pickup without requiring repeated wake-ups. The continuous dialogue continues until the user exits via a command such as "exit," or by clicking the back control / back button / exit control / exit button, or by performing a preset back gesture on the display screen.

[0061] What is visible can be said:

[0062] It can be seen that this is achieved locally by electronic devices and does not require a network connection.

[0063] The "Visible and Speakable" feature is controlled by a preset switch. Users can turn on the preset switch to enable "Visible and Speakable" or turn it off to disable it.

[0064] Taking a mobile phone as an example, for instance, such as Figure 3 As shown, users can access their phone's settings; for example, by clicking the "Settings" app icon on the home screen. In response to the user's click on the "Settings" app icon, the phone displays the "Settings" interface 101. The "Settings" interface 101 includes a "Smart Voice" option 102, which is used to configure the smart voice function. For example, in response to the user's click on the "Smart Voice" option 102, the phone displays the "Smart Voice" interface 103, which includes a "See and Speak" option 104. Users can click on the "See and Speak" option 104 to configure options related to the "See and Speak" function.

[0065] For example, refer to Figure 3In response to a user's click on the "Speak When You See It" option 104, the phone displays the "Speak When You See It" interface 105. Optionally, the "Speak When You See It" interface 105 includes a prompt message 106 to instruct the user on how to use the "Speak When You See It" function. The "Speak When You See It" interface 105 also includes a "Speak When You See It" switch 107. The user can click the "Speak When You See It" switch 107 to turn the "Speak When You See It" switch on or off. In one example, in response to receiving a user's click on the "Speak When You See It" switch 107, the phone's "Speak When You See It" switch is turned on, enabling the "Speak When You See It" function. Optionally, the "Speak When You See It" interface 105 displays a prompt message 108 to inform the user that the "Speak When You See It" function has been successfully enabled.

[0066] In one implementation, after the "See It, Say It" function is enabled, the phone displays a first notification icon indicating that the "See It, Say It" function is enabled. For example, such as... Figure 3 As shown, after the "See and Say" switch 107 is turned on, the status bar of the phone's display screen shows a prompt icon 10a, indicating that the "See and Say" function has been enabled.

[0067] In one scenario, the preset switch for "See and Say" on the electronic device is not turned on, and the device's microphone is not activated. When the preset switch for "See and Say" is turned on, the electronic device activates the microphone and starts the corresponding recording channel in the system and driver. In this way, the voice (audio stream) captured by the microphone can be sent to the "See and Say" application for processing through the corresponding recording channel, enabling the user to control the electronic device via voice.

[0068] In another scenario, the preset switch for "Visible & Talkable" on the electronic device is not turned on. The device's microphone operates in a power-saving mode (e.g., searching for signals at low power) to pick up ambient sound. When the preset switch for "Visible & Talkable" is turned on, the corresponding recording channel for "Visible & Talkable" is activated in the device's system and drivers. In this way, the voice (audio stream) captured by the microphone can be sent to the "Visible & Talkable" application for processing through the corresponding recording channel, enabling the user to control the electronic device via voice.

[0069] Once "Visible and Talkable" is enabled, the corresponding recording channel is activated in the electronic device's system and drivers. The electronic device enters a "long-reception" state, continuously capturing ambient sound. Users can issue commands to the electronic device at any time via voice, without needing to input a wake-up word.

[0070] In one implementation, after any function on the electronic device activates its voice recording function (turns on the microphone and establishes a recording channel), the electronic device will send a prompt message to the user, indicating that the device is in voice recording mode. This helps prevent the leakage of user privacy. For example, after the "see and speak" function is activated, the electronic device enters continuous voice recording mode, and a second prompt icon is displayed on the device's screen. This second prompt icon indicates that the recording channel is open, indicating to the user that the microphone is recording voice. For example, such as... Figure 3 As shown, the status bar on the phone's screen displays a notification icon 10b, indicating that the recording channel is enabled and is used to notify the user that the microphone is capturing voice.

[0071] Once the "See It, Speak It" function is enabled, the electronic device establishes a recording channel and continuously captures the user's voice through the microphone. The user can input voice into the electronic device at any time. The electronic device parses and recognizes the user's voice input and executes the commands corresponding to the user's voice.

[0072] In one example, such as Figure 4 As shown, the phone displays desktop interface 120. The user inputs the voice command "Open Calendar" into the phone. In response to receiving the user's voice command "Open Calendar," the phone launches the calendar application and displays the calendar application's user interface 121.

[0073] In yet another example, such as Figure 5 As shown, the phone displays the interface 123 of a short video application. The user inputs the voice command "pause" into the phone. In response to receiving the user's voice command "pause," the phone executes the corresponding command and displays the short video application interface 124. This interface 124 displays a "play" button 125 to indicate to the user that the short video is paused. The user can click the "play" button 125 to resume playing the short video. Alternatively, the user can input the voice command "play" into the phone to resume playing the short video.

[0074] VoiceKit: It can be integrated with voice control functions within the same application. Some voice control functions can interface with the ASR engine through VoiceKit. For example, "See and Say" and the voice assistant can each interface with the ASR engine through VoiceKit.

[0075] The ASR engine can convert speech signals into text or commands based on machine recognition and understanding.

[0076] Handler: This is a parameter used by the ASR engine to represent the voice control function currently using the ASR engine. After being invoked and initialized by a voice control function, the ASR engine can generate a corresponding handler based on the voice control function that initiated the call request. Different voice control functions may call the ASR engine, resulting in different handlers. Conversely, the same voice control function may call the ASR engine, generating the same handler.

[0077] Natural Language Understanding (NLU) is a general term for all methods, models, or tasks that support machines in understanding text content.

[0078] Dialogue Management (DM) is used to control the process of human-computer dialogue and determine the current response to the user based on dialogue history information.

[0079] It is now possible for users to control electronic devices via voice. For example, in scenarios like cooking and watching videos in the kitchen, or eating and watching TV, a user's hands may be occupied. In these situations, users typically control the electronic device by speaking voice commands. Users within a certain range of the phone can control it by speaking voice commands. A single electronic device may have two or more different voice control functions. These different voice control functions can be used in different scenarios to control the electronic device to perform different operations and achieve different control effects.

[0080] In some electronic devices, different voice control functions can share the same ASR engine. However, in related technologies, the ASR engine in electronic devices is typically a single instance, meaning only one service can access and use it at a time, and concurrency is not supported. Furthermore, different voice control functions may support different voice commands, and the context and hot words used during speech recognition may also differ. Therefore, it is also clear that two different voice control functions cannot use the same ASR engine simultaneously.

[0081] Therefore, during the process of voice control function 1 using the ASR engine, if voice control function 2 requests to call the ASR engine, under certain conditions, the ASR engine usually needs to switch from providing services to voice control function 1 to providing services to voice control function 2.

[0082] Taking voice control function 1 ("See and Say") and voice control function 2 ("Voice Assistant") as examples, as explained above, both "See and Say" and the voice assistant can be connected to the ASR engine through VoiceKit. Please refer to... Figure 6 The process shown involves the mobile phone responding to a trigger operation for "Visible and Talkable" by first launching "Visible and Talkable". "Visible and Talkable" then calls the initialization-related interfaces in the ASR engine via VoiceKit, causing the ASR engine to initialize the service data to correspond with the service data of "Visible and Talkable" and provide services to it. In some embodiments, the initialization of service data by the ASR engine to correspond with the service data of "Visible and Talkable" can also be referred to as the ASR engine establishing a correspondence with "Visible and Talkable".

[0083] Afterwards, the phone responds to the trigger operation on the voice assistant and can launch it. Since the ASR engine is already occupied by VisibleSay, the voice assistant needs to first notify VisibleSay to deregister the ASR engine. Specifically, VisibleSay uses VoiceKit to call the deregistration-related interface in the ASR engine to deregister the corresponding service data of VisibleSay, thus releasing the ASR engine. During the process of VisibleSay deregistering its corresponding service data via VoiceKit, VoiceKit will also be deregistered.

[0084] As can be seen, after releasing the ASR engine, the voice assistant can use VoiceKit to call the initialization-related interfaces in the ASR engine, initializeing the ASR engine's service data to the voice assistant's corresponding service data. Then, the voice assistant can exit in response to an exit operation. At this point, the voice assistant will call the ASR engine's deregistration interface through VoiceKit to achieve the effect of releasing the ASR engine.

[0085] Since VisibleSpeaker remains active when the voice assistant exits, the phone needs to reassign the ASR engine to VisibleSpeaker after the voice assistant exits to restore voice control functionality based on VisibleSpeaker. The voice assistant exits recording, causing a change in the recording state. Therefore, in some embodiments, VisibleSpeaker can detect the voice assistant's exit through the change in the recording state. This allows VisibleSpeaker to call the ASR engine's initialization interface again via VoiceKit to reinitialize the ASR engine and provide services to VisibleSpeaker.

[0086] In the scenario described above, where "Visible and Speakable" starts first, followed by the voice assistant, then exits and reverts to "Visible and Speakable," the ASR engine needs to be released and initialized multiple times sequentially by both the visible and speakable components to allow the ASR engine to switch between them. Furthermore, the process of releasing and initializing the ASR engine requires repeated unregistration and initialization of VoiceKit.

[0087] However, VoiceKit includes multiple modules such as data acquisition, data processing, intent understanding, intent execution, and capability connectors, and is also bound to engines such as ASR, NLU, and DM through capability connectors. During the aforementioned processes of "See-It-Speak" and voice assistant deregistration and VoiceKit initialization, not only are the modules within VoiceKit responsible for interfacing with the ASR engine repeatedly unbound and rebound, but other modules within VoiceKit also repeatedly unbound and rebound with other engines. However, apart from the ASR engine, the usage process of "See-It-Speak" and voice assistants for other engines is the same. Therefore, in scenarios where "See-It-Speak" starts, the voice assistant starts and then exits, and then "See-It-Speak" is restored, the repeated deregistration and initialization of VoiceKit, including the repeated unbound and rebound processes of modules other than the ASR engine, is wasteful. This may increase latency during service switching and cause unnecessary power consumption.

[0088] Based on this, this application provides a method for invoking a speech recognition engine, which can be applied to electronic devices. The electronic device has a first voice control function, and a first voice control module corresponding to the first voice control function interfaces with an ASR engine via VoiceKit. In response to a first event, when VoiceKit is in the started state, the first voice control module does not need to reinitialize VoiceKit, but can directly initiate a first call request to the ASR engine for the initialization interface through VoiceKit. Subsequently, in response to the first call request, the ASR engine will initialize the first service data corresponding to the first voice control module. Afterwards, the ASR engine will provide services for the first voice control function.

[0089] In this solution, when the first voice control function needs to call the ASR engine, if the voicekit is detected to be running, it will not be deregistered and reinitialized. Instead, the voicekit can directly call the ASR engine's initialization interface to achieve the effect of the first voice control function calling the ASR engine. This avoids repeated deregistration and initialization of the voicekit in scenarios where two or more voice control functions using the same ASR engine are launched sequentially. This prevents repeated unbinding and binding between other modules within the voicekit (besides the ASR engine) and other engines. Consequently, it reduces latency during ASR engine service switching and minimizes unnecessary power consumption.

[0090] The speech recognition engine invocation method provided in this application embodiment can be applied to an electronic device 100 including a microphone. The aforementioned electronic device 100 may include mobile phones, tablets, laptops, personal computers (PCs), ultra-mobile personal computers (UMPCs), handheld computers, netbooks, smart home devices (e.g., smart TVs, smart screens, large screens, smart speakers, smart air conditioners, etc.), personal digital assistants (PDAs), wearable devices (e.g., smartwatches, smart bracelets, etc.), in-vehicle devices, virtual reality devices, etc., and this application embodiment does not impose any limitations on these.

[0091] like Figure 7 The diagram shown is a structural schematic of an electronic device 100 provided in an embodiment of this application. The electronic device 100 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, antenna 1, antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a sensor module 280, buttons 290, a motor 291, a camera 292, a display screen 293, and a subscriber identification module (SIM) card interface 294, etc. The sensor module 280 may include a pressure sensor 280A, a touch sensor 280B, etc.

[0092] It is understood that the structures illustrated in the embodiments of this application do not constitute a specific limitation on the electronic device 100. In other embodiments of this application, the electronic device 100 may include more or fewer components than illustrated, or combine some components, or split some components, or have different component arrangements. The illustrated components may be implemented in hardware, software, or a combination of software and hardware.

[0093] Processor 210 may include one or more processing units, such as an application processor (AP), a modem processor, a graphics processing unit (GPU), an image signal processor (ISP), a controller, memory, a video codec, a digital signal processor (DSP), a baseband processor, and / or a neural network processing unit (NPU). Different processing units may be independent devices or integrated into one or more processors. For example, processor 210 is used to execute the invocation method of the speech recognition engine in this embodiment.

[0094] The controller can be the nerve center and command center of the electronic device 100. The controller can generate operation control signals according to the instruction opcode and timing signals to complete the control of fetching and executing instructions.

[0095] The processor 210 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. This memory can store instructions or data that the processor 210 has just used or that are used repeatedly. If the processor 210 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0096] USB interface 230 is an interface that conforms to the USB standard specification, specifically it can be a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 230 can be used to connect a charger to charge electronic device 100, and it can also be used for data transfer between electronic device 100 and peripheral devices.

[0097] The external storage interface 220 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 210 through the external storage interface 220 to perform data storage functions. For example, music, video, and other files can be saved on the external memory card.

[0098] Internal memory 221 can be used to store executable program code, which includes instructions. Processor 210 executes various functional applications and data processing of electronic device 100 by running the instructions stored in internal memory 221. Internal memory 221 may include a program storage area and a data storage area. The program storage area may store the operating system and at least one application program required for a given function (such as sound playback, image playback, etc.).

[0099] In addition, the internal memory 221 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0100] The charging management module 240 is used to receive charging input from the charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 can receive charging input from the wired charger via the USB interface 230.

[0101] The power management module 241 is used to connect the battery 242, the charging management module 240, and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, internal memory 221, external memory, display 293, camera 292, and wireless communication module 260, etc.

[0102] In some other embodiments, the power management module 241 may also be located within the processor 210. In other embodiments, the power management module 241 and the charging management module 240 may also be located in the same device.

[0103] The wireless communication function of electronic device 100 can be implemented through antenna 1, antenna 2, mobile communication module 250, wireless communication module 260, modem processor, and baseband processor.

[0104] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with tuning switches.

[0105] The mobile communication module 250 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the electronic device 100. The mobile communication module 250 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1.

[0106] The wireless communication module 260 can provide solutions for wireless communication applications on the electronic device 100, including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth, Global Navigation Satellite System (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR). The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signal, and sends the processed signal to processor 210. The wireless communication module 260 can also receive signals to be transmitted from processor 210, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2.

[0107] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, enabling electronic device 100 to communicate with networks and other devices via wireless communication technology.

[0108] Electronic device 100 can implement audio functions, such as music playback and recording, through audio module 270 and application processor.

[0109] The audio module 270 is used to convert digital audio signals into analog audio signals for output, and also to convert analog audio inputs into digital audio signals. The audio module 270 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 270 may be located in the processor 210, or some functional modules of the audio module 270 may be located in the processor 210.

[0110] Pressure sensor 280A is used to sense pressure signals and convert them into electrical signals. In some embodiments, pressure sensor 280A can be disposed on display screen 293. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When a force is applied to pressure sensor 280A, the capacitance between the electrodes changes. Electronic device 100 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to display screen 293, electronic device 100 detects the intensity of the touch operation based on pressure sensor 280A. Electronic device 100 can also calculate the touch position based on the detection signal from pressure sensor 280A.

[0111] Touch sensor 280B, also known as a "touch panel," can be located on display screen 293. The touch sensor 280B and display screen 293 together form a touchscreen, also known as a "touch screen." Touch sensor 280B detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through display screen 293. In other embodiments, touch sensor 280B may also be located on the surface of electronic device 100, in a different position than display screen 293.

[0112] Buttons 290 include a power button, volume buttons, etc. Buttons 290 can be mechanical buttons or touch-sensitive buttons. Electronic device 100 can receive button input and generate key signal inputs related to user settings and function control of electronic device 100.

[0113] Motor 291 can generate vibration alerts. Motor 291 can be used for incoming call vibration alerts or for touch vibration feedback.

[0114] Electronic device 100 implements display functions through a GPU, display screen 293, and application processor. The GPU is a microprocessor for image processing, connected to the display screen 293 and the application processor. The GPU is used to perform mathematical and geometric calculations and for graphics rendering. Processor 210 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0115] The display screen 293 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include one or N display screens 293, where N is a positive integer greater than 1.

[0116] Camera 292 is used to capture still images or videos. In some embodiments, electronic device 100 may include one or N cameras 292, where N is a positive integer greater than 1.

[0117] The SIM card interface 294 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 294 to make contact with or separate from the electronic device 100. The electronic device 100 can support one or N SIM card interfaces, where N is a positive integer greater than 1.

[0118] The methods for invoking the speech recognition engine in the following embodiments can all be implemented in the electronic device 100 with the above-described hardware structure.

[0119] In this embodiment, the electronic device 100 is an electronic device capable of running an operating system and installing applications. Optionally, the operating system running on the electronic device may be... system, system, Systems, etc.

[0120] In some embodiments, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a microkernel architecture, or a cloud architecture. This application embodiment uses a layered architecture. Taking the system as an example, the software structure of the electronic device 100 is illustrated.

[0121] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, [the following is omitted as the text is incomplete and likely refers to a specific implementation or feature]. The system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer.

[0122] like Figure 8 As shown, the application layer may include a series of application packages (APKs). Examples include applications for camera, recording, gallery, calendar, calling, map, navigation, WLAN, Bluetooth, music, video calling, and SMS. The application layer also includes a voice processing engine for parsing, recognizing, and processing voice. In this embodiment, the application layer includes a voice control APK for providing voice control functions (e.g., visible-to-speak function, voice assistant function) for the electronic device 100. For example, the voice control APK includes a visible-to-speak APK and a voice assistant application; the visible-to-speak APK provides the visible-to-speak function of the electronic device 100, and the voice assistant application provides the voice assistant function of the electronic device 100; wherein, the visible-to-speak APK includes a business presentation module, a business processing module, a voice processing module, etc.

[0123] The business presentation module provides information displayed to the user on the user interface. For example, it provides prompts displayed on the electronic device 100's display screen. The business processing module controls the process of recording voice input via the "Visible and Talkable" function. For example, it controls the enabling and disabling of the "Visible and Talkable" function, and whether the function is available or unavailable. The business processing module may include a recording management module, a status management module, and a monitoring module. The monitoring module monitors whether other applications are using the microphone. The recording management module controls the enabling and disabling of the recording channel corresponding to the "Visible and Talkable" function; for example, it enables or disables the recording channel based on the "Visible and Talkable" switch. The status management module manages the status of the "Visible and Talkable" function, such as whether it is enabled, disabled, or temporarily unavailable.

[0124] The voice processing module is used to recognize and process the collected user voice. For example, the voice processing module calls the voice processing engine to perform text parsing, semantic understanding, and recognition on the audio stream captured by the microphone, and converts the user's voice into voice commands. The voice processing module sends the voice commands to the multi-modal control module in the application framework layer, and the multi-modal control module distributes the voice commands to the corresponding applications for execution.

[0125] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 8 As shown, the application framework layer may include a window manager, content provider, view system, resource manager, activity manager service (AMS), application manager service (PMS), multi-mode control module, audio module, etc.

[0127] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0128] Content providers store and retrieve data, making that data accessible to applications. This data can include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, and more.

[0129] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0130] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0131] AMS is primarily responsible for starting, switching, and scheduling the four main components of the system, as well as managing and scheduling application processes. Its responsibilities are similar to those of the process management and scheduling module in an operating system. When a process or component starts, the request is passed to AMS via the Binder communication mechanism, and AMS then processes it uniformly.

[0132] PMS handles package management tasks, such as application installation and uninstallation. It also provides comprehensive information about an application, including its startup and shutdown status.

[0133] The multi-mode control module is used to manage the execution of voice commands. For example, it sends voice commands to applications, causing the applications to execute those commands.

[0134] The audio module handles audio-related functions, such as recording and playing audio. This includes a microphone, which manages the audio recording process.

[0135] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0136] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0137] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0138] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0139] A 2D graphics engine is a graphics engine for 2D drawing.

[0140] The Android runtime is responsible for scheduling and managing the Android system. The Android runtime includes core libraries and a virtual machine.

[0141] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0142] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0143] The kernel layer is the layer between hardware and software. It can contain display drivers, sensor drivers, audio drivers, Wi-Fi drivers, etc. The kernel layer also includes a wake-word detection module, used to detect whether a wake-word has been received based on the audio captured by the microphone. Furthermore, the kernel layer includes an audio digital signal processor (ADSP), used to distribute and process the audio captured by the microphone, such as establishing recording channels for applications and distributing audio streams to applications.

[0144] Combination Figure 8 , Figure 9 The diagram illustrates the interaction of various modules of an electronic device when implementing the voice-controlled electronic device provided in the embodiments of this application.

[0145] Please see Figure 9 It is evident that both iTalk and the voice assistant require connection to the voice control engine via VoiceKit. Specifically, iTalk and the voice assistant can interface with the data acquisition, data understanding, intent understanding, and intent processing modules within VoiceKit, and are bound to the ASR, NLU, and DM engines within the voice control engine through the capability connectors in VoiceKit. Therefore, iTalk and the voice assistant can utilize all the engines within the voice control engine to achieve related speech recognition and control functions.

[0146] The following combination Figure 9Taking "See-It-All-Speak" as an example, the process of voice control of electronic devices based on "See-It-All-Speak" is explained as follows: When using "See-It-All-Speak" to control electronic devices, the screen content module of "See-It-All-Speak" obtains screen elements from the frame layer for text analysis and layout analysis, and then transmits the analyzed screen context to the command distribution module. After acquiring audio data, "See-It-All-Speak" can transmit the audio data to the voice control engine through VoiceKit for data understanding, intent understanding, and intent processing, thereby determining the voice command to be executed in the audio data. Then, the voice control engine transmits the determined voice command to the command distribution module of "See-It-All-Speak," which determines the type of voice command (control click type or general command type). Then, the command distribution module can transmit the determined voice command to the command orchestration and execution module for preprocessing, adaptation, or conversion. Finally, the command orchestration and execution module transmits the voice command to the multi-module interaction framework, and the corresponding operation of the voice command is executed through the multi-module interaction framework.

[0147] The following will describe in detail the method for invoking the speech recognition engine provided in the embodiments of this application, with reference to the accompanying drawings. The following description uses a mobile phone as an example, where the mobile phone first activates a second voice control function and then activates a first voice control function. Furthermore, in this embodiment, the second voice control function is "visible and speakable," and the first voice control function is a voice assistant. In this embodiment, the mobile phone may include: a control module corresponding to "visible and speakable," a control module corresponding to a voice assistant, a voice kit, and an ASR engine. Please refer to... Figure 10 The method includes:

[0148] S401. Visible and speakable receive voice control trigger event 1.

[0149] Voice control trigger event 1 is used to trigger the second voice control function (in this embodiment, it is visible and can be spoken) to start.

[0150] In some embodiments, the aforementioned voice control trigger event 1 can specifically be a visible-to-speak activation event. Specifically, the visible-to-speak activation event can correspond to the user manually activating the visible-to-speak control function. For example, the user can trigger an event such as... Figure 3 The switch 107 in the settings interface 105 shown is used to turn on the visible-to-talk control function.

[0151] In other embodiments, the aforementioned voice control trigger event 1 can also be the visible or verbal detection of another voice function exiting. This voice function can be a voice assistant or other voice control function, an audio / video call, or a recording function, etc. Combined with... Figure 6As can be seen from the process, after "See and Speak" is activated, if the user triggers the voice assistant, the voice control function based on "See and Speak" will be temporarily interrupted. After the voice assistant exits, "See and Speak" can resume voice control. Therefore, in this embodiment, the aforementioned voice control trigger event 1 can also be a voice assistant exit event. It is understandable that in other embodiments, "See and Speak" may also resume after voice control is interrupted by other voice functions; in this embodiment, voice control trigger event 1 can specifically be the exit event of the corresponding voice function.

[0152] exist Figure 10 In the embodiment shown, the phone has not enabled "See and Speak" before S401. Therefore, the voice control trigger event 1 in S401 is the "See and Speak" activation event.

[0153] S402. In response to voice control trigger event 1, activate the visible and speakable voice control function.

[0154] In some embodiments, during the activation of the "See and Speak" function, the phone will establish a corresponding recording channel. The audio stream subsequently captured by the microphone will be distributed to the "See and Speak" application for processing through the recording channel corresponding to the "See and Speak" function. Furthermore, after establishing the recording channel corresponding to "See and Speak," the "See and Speak" function will invoke the voice control engine to enable voice control based on "See and Speak."

[0155] S403. It can be seen that a call request is sent through the voicekit's initialization interface for the ASR engine.

[0156] As explained above, Visible and Speakable needs to interface with the ASR engine through VoiceKit. Therefore, when Visible and Speakable requests to invoke the ASR engine, it needs to send a request to the ASR engine's initialization interface through VoiceKit.

[0157] It should be noted that when a mobile phone implements voice control based on "Visible and Speakable," it needs to use engines such as the ASR engine, NLU engine, and DM engine in the voice control function. Therefore, in some embodiments, after "Visible and Speakable" is started, in addition to sending call request 1 to the ASR engine through VoiceKit, it also needs to send call requests to other engines in the voice control engine besides the ASR engine through VoiceKit.

[0158] In this embodiment, the example is taken before the ASR engine is started. That is, before the Visible and Speakable requests to invoke the ASR engine, Voicekit is not started. Therefore, after the Visible and Speakable sends call request 1 to Voicekit, Voicekit itself needs to be initialized first, as in S404.

[0159] S404. In response to call request 1, voicekit is initialized.

[0160] The specific implementation process of voicekit initialization can be found in the descriptions in related technologies, and will not be repeated in the embodiments of this application.

[0161] S405.voicekit sends a call request to the ASR engine for the ASR engine's initialization interface.

[0162] After initialization is complete, VoiceKit can send a call request to the ASR engine's initialization interface. In some embodiments, VoiceKit can forward a "see-is-speak" call request to the ASR engine.

[0163] In some embodiments, the initialization interface of the ASR engine can also be called the init interface.

[0164] The S406.ASR engine responds to call request 1 by binding with VoiceKit.

[0165] Combination Figure 9 It is known that the ASR engine is bound to VoiceKit, which can specifically include: the module in VoiceKit responsible for interfacing with the ASR engine is bound to the ASR engine through a capability connector.

[0166] The S407.ASR engine initializes the service data to the visible and speakable service data 1.

[0167] Since call request 1 is initiated to the ASR engine's initialization interface, the ASR engine will initialize in response to call request 1. In some embodiments, S407 may specifically include: registering the context, hot words, etc., corresponding to the "See and Speak" control function in the ASR engine. This allows the ASR engine to combine the context and hot words to perform speech recognition on the audio data sent after "See and Speak" and determine the voice commands therein. It is understood that after the ASR engine initializes the service data 1 corresponding to "See and Speak," the ASR engine can provide services for "See and Speak."

[0168] After initializing in response to a voice control function call request, the ASR engine can write relevant information about the voice control function currently calling the ASR engine into the phone's cache. In some embodiments, the relevant information about the voice control function written into the cache by the ASR engine may specifically include: function identification information of the voice control function (such as ID, corresponding package name, etc.). In this embodiment, S407 above may specifically include: the ASR engine writing relevant information that is visible and can be spoken into the cache.

[0169] Furthermore, in some embodiments, after writing relevant information about the voice control function currently invoking the ASR engine into the cache, the ASR engine can also generate corresponding call parameters based on that voice control function. It is understood that these call parameters can be used to characterize the voice control function currently being used by the ASR engine, i.e., the current user of the ASR engine. In some embodiments, after writing the "visible and speakable" relevant information into the cache, the ASR engine also generates call parameters corresponding to "visible and speakable" and saves these call parameters as the current call parameters of the ASR engine. In some embodiments, the aforementioned call parameters may specifically be handlers.

[0170] S408. Voice assistant receives voice control trigger event 2.

[0171] In this embodiment of the application, the first voice control function is a voice assistant; therefore, the aforementioned voice control trigger event 2 is the first voice control trigger event.

[0172] In some embodiments, the aforementioned voice control trigger event 2 may specifically be a voice assistant launch event. The voice assistant launch event may correspond to a user activating the voice assistant control function. For example, the voice assistant launch event may specifically correspond to any of the following events: the phone detects a preset operation on a preset button, or the phone detects a wake word for the voice assistant, or the phone detects a trigger operation on the voice assistant function icon.

[0173] Similar to the first event, after the voice assistant is activated, it may be interrupted by other voice functions (such as voice control functions like "see and speak," audio / video calls, or recording functions). Afterwards, once the other voice functions exit, the voice assistant can resume voice control. Therefore, in some embodiments, the aforementioned voice control trigger event 2 can also correspond to the exit event of other voice functions.

[0174] exist Figure 10 In the embodiment shown, the phone has not started the voice assistant before S408. Therefore, the voice control trigger event 2 in S408 is the voice assistant start event.

[0175] S409. In response to voice control trigger event 2, activate the voice control function of the voice assistant.

[0176] In some embodiments, during the activation of the voice assistant, the phone establishes a recording channel corresponding to the voice assistant function. The audio stream subsequently captured by the microphone is then distributed to the voice assistant application for processing through the recording channel corresponding to the voice assistant function.

[0177] S410. The voice assistant sends a call request to the ASR engine's initialization interface via VoiceKit.

[0178] In this embodiment of the application, the first voice control function is a voice assistant; therefore, the above-mentioned call request 2 is the first call request.

[0179] It should be noted that when a mobile phone implements voice control based on a voice assistant, the voice assistant needs to use engines such as the ASR engine, NLU engine, and DM engine in the voice control function. Therefore, in some embodiments, after the voice assistant is started, in addition to sending call request 2 to the ASR engine through VoiceKit, it also needs to send call requests to other engines in the voice control engine besides the ASR engine through VoiceKit.

[0180] In response to call request 2, the S411.ASR engine unregisters the visible and speakable service data 1 and initializes the service data to the service data 2 corresponding to the voice assistant.

[0181] In this embodiment, the first voice control function is a voice assistant; therefore, the aforementioned service data 2 is the first service data. In some embodiments, the ASR engine can unregister the service data 1 corresponding to "Visible and Speakable," thereby releasing the ASR engine.

[0182] Understandably, after the ASR engine initializes its service data to the service data 2 corresponding to the voice assistant, the ASR engine can then provide services to the voice assistant.

[0183] Furthermore, since the ASR engine in S411 cancels the service data corresponding to "Visible and Speakable" in response to other functions' calls to the initialization interface, rather than in response to "Visible and Speakable" itself's call to the ASR engine's cancellation interface, in some embodiments, after the ASR engine switches from "Visible and Speakable" to the voice assistant in response to call request 2, the ASR engine can suspend "Visible and Speakable".

[0184] In some embodiments, when the ASR engine cancels the service data 1 corresponding to "Visible and Speakable," it does not delete the relevant information about "Visible and Speakable" stored in the ASR engine's cache. This is so that after the voice assistant exits, the ASR engine can respond to a new call request from "Visible and Speakable" and restore the call to the ASR engine based on the relevant information stored in the cache. The specific process by which the ASR engine restores the call to the ASR engine in response to a new call request from "Visible and Speakable" after the voice assistant exits will be described in detail in later embodiments.

[0185] Meanwhile, in some embodiments, after the ASR engine initializes the service data to the service data 2 corresponding to the voice assistant in S411, the above method may also include: the ASR engine inserting relevant information of the voice assistant into the cache.

[0186] Furthermore, although the ASR engine does not delete the visible-to-speak information from the cache, since the voice control function currently calling the ASR engine has switched to a voice assistant, the ASR engine needs to generate new call parameters based on the voice assistant. In some embodiments, after inserting the voice assistant's relevant information into the cache, the ASR engine will also generate new call parameters based on the corresponding voice assistant. In some embodiments, the ASR engine saves the latest call parameters in the cache; that is, the ASR engine updates the current call parameters from the second call parameter corresponding to the visible-to-speak information to the first call parameter corresponding to the voice assistant.

[0187] In some embodiments, the ASR engine can cache information about the current user in a state queue. For example, the ASR engine can insert information about the voice assistant or other visible and speakable features at the head of the state queue.

[0188] Understandably, in the above S411, the ASR engine initializes the ASR engine's service data with the service data corresponding to the voice assistant. Specifically, this can be done after the ASR engine has completed the unregistration of the service data corresponding to what is visible and what can be spoken.

[0189] In S410, when the ASR engine receives call request 2 sent by the voice assistant via VoiceKit, the ASR engine is currently providing services for "what is visible can be spoken". That is, the ASR engine is currently in a occupied state. In some embodiments, if the ASR engine receives a call request from another function during use, it can first determine whether the switching conditions are met. If so, the ASR engine is allowed to switch from the current function to the other function; otherwise, switching is not allowed.

[0190] In some embodiments, after the ASR engine receives call request 2 in S410 and before S411, the method further includes: determining whether a preset switching condition is met. In this embodiment, S411 may specifically include: if the switching condition is met, in response to call request 2, the ASR engine deregisters the second access data corresponding to the visible / speakable function and initializes the service data to the service data 2 corresponding to the voice assistant.

[0191] Furthermore, in some embodiments, the ASR engine determines whether the switching conditions are met. Specifically, this may include determining whether the voice control function currently using the ASR engine (such as "see and speak") allows switching. If "see and speak" allows switching, the switching conditions are met; in this case, the ASR engine can log out and reload. If "see and speak" does not allow switching, the switching conditions are not met; in this case, the ASR engine can directly ignore subsequent call requests, or the ASR engine can return an initialization failure response message to the function that initiated the call request later (such as a voice assistant).

[0192] Alternatively, the mobile phone can pre-set a switchable whitelist of allowed functions in the ASR engine. This whitelist can be used to define the functions that can be switched. In this embodiment, the ASR engine determines whether the switching conditions are met, specifically by determining whether the currently used ASR engine function is in the switchable whitelist. Further, in this embodiment, S411 specifically includes: if the visible / speakable function is included in the switchable whitelist, in response to call request 2, the ASR engine deregisters the visible / speakable service data 1 and initializes the service data to the service data 2 corresponding to the voice assistant.

[0193] Alternatively, for multiple functions that support calling the ASR engine, the mobile phone can pre-set priorities and store this pre-set priority information in the ASR engine. Functions with higher priorities can call the ASR engine first. Therefore, in some embodiments, the ASR engine's determination of whether the switching conditions are met may also include: the ASR engine can determine, based on the pre-stored priority information, whether the priority of the function that initiates the call request later (such as the voice assistant) is higher than that of the function that initiates the call request earlier (such as "See and Speak"). Further, in this embodiment, S411 may specifically include: if the voice assistant's priority is higher than the "See and Speak" priority, then in response to call request 2, the ASR engine deregisters the service data 1 corresponding to "See and Speak" and initializes the service data to the service data 2 corresponding to the voice assistant.

[0194] In other embodiments, if the function that initiates the call request later has a lower priority than the function that initiates the call request earlier, the ASR engine may ignore the later-initiated call request; or, the ASR engine may return an initialization failure response message to the function that initiates the call request later. For example, if the ASR engine determines that the voice assistant's priority is lower than the priority of "visible and speakable," the ASR engine may ignore call request 2 sent by the voice assistant through VoiceKit; or, the ASR engine may return an initialization failure response message to the voice assistant.

[0195] In the technical solution provided in this application embodiment, for scenarios where the ASR engine receives a new call request while in use, the ASR engine will only allow the switch from the currently using ASR engine to the function that initiated the call request if the switching conditions are met. This avoids accidental triggering and switching, preventing the voice control function from properly using the ASR engine.

[0196] S412. In response to the first exit event, the voice assistant exits and sends a call request to the ASR engine for the ASR engine's logout interface via voicekit 3.

[0197] In some embodiments, the ASR engine's deregistration interface can also be referred to as the ASR engine's destroy interface.

[0198] In some cases, users may accidentally trigger the voice assistant. In such situations, after realizing that the voice assistant has been activated, the user can manually exit it. For example, the user can exit the voice assistant by clicking the back control / back button / exit control / exit button, or by performing a preset back gesture on the display screen. Therefore, in some embodiments, the first exit event specifically corresponds to the phone detecting the user's exit operation on the voice assistant.

[0199] Alternatively, after the voice assistant is activated, the user can also control the phone to exit the voice assistant via voice control. For example, after the voice assistant is activated, the user can say a voice command such as "exit" to control the phone to exit the voice assistant; the voice assistant will exit in response to the voice command. Therefore, in some other embodiments, the first exit event can also correspond to the detection of a voice command for the exit operation.

[0200] Alternatively, some mobile phones can also be equipped with an automatic exit mechanism for the voice assistant. For example, if the voice assistant does not receive valid audio data within a certain period of time after it is started, it can automatically exit. In other embodiments, the first exit event can also correspond to the voice assistant exiting when no valid audio data is received within a preset time period.

[0201] It is understood that the above embodiments are only some examples of the first exit event, and in other embodiments, the first exit event may be other events.

[0202] In response to call request 3, the S413.ASR engine deregisters the service data 2 corresponding to the voice assistant.

[0203] As shown in S412, call request 3 is sent by the voice assistant itself to the ASR engine via VoiceKit. Therefore, in some embodiments, in response to call request 3, the ASR engine can delete the relevant information of the voice assistant from the cache while canceling the service data 2 corresponding to the voice assistant.

[0204] Furthermore, in response to the first exit event, the voice assistant exits, meaning it will cease using recording permissions. Therefore, in some embodiments, a recording status change event, such as S414, will also be triggered.

[0205] S414. The voice assistant triggers a recording status change event.

[0206] Understandably, the mobile phone can execute S412-S413 first, then S414, or it can execute S414 first, then S412-S413, or it can execute S412-S413 and S414 simultaneously. In this embodiment, the order of execution of S412-S413 and S414 is not limited.

[0207] As described in the above embodiments, the recording status change event in S414 is triggered by the voice assistant exiting. This recording status change event indicates that the voice assistant has stopped using recording permissions. Simultaneously, the voice assistant exiting also indicates that the application assistant will stop using the ASR engine. Referring to the descriptions of S401-S407, VisibleSpeaker is started before the voice assistant starts; and after the voice assistant starts, VisibleSpeaker is temporarily interrupted. That is, during voice control based on the voice assistant, VisibleSpeaker is in a state suspended by the ASR engine. Therefore, when the voice assistant exits, VisibleSpeaker can resume using the ASR engine to restore the voice control function based on VisibleSpeaker.

[0208] S415. It can be seen that, in response to the recording state change event, a call request for the ASR engine's initialization interface is sent to the ASR engine via VoiceKit.

[0209] As described in the above embodiments, the voice assistant triggers a voice state change event upon exiting. Therefore, it can be seen that after detecting the recording state change event, it can be determined that the voice assistant has exited. Afterwards, it can be seen that the ASR engine can be invoked again. In this embodiment, the recording state change event belongs to the aforementioned voice control trigger event 1. Therefore, in response to this recording state change event, it can be seen that the aforementioned invocation request 4 can be sent to the ASR engine via VoiceKit.

[0210] Therefore, when sending a call request to the ASR engine again through VoiceKit, if VoiceKit is already running, then the call request can be sent directly through VoiceKit. If VoiceKit is not running, then VoiceKit needs to be initialized first before sending a call request to the ASR engine.

[0211] In some embodiments, upon detecting a recording status change event, VisibleSpeak can first determine whether it has previously invoked the ASR engine to ascertain the voicekit's status. If VisibleSpeak has previously invoked the ASR engine, then the voicekit should be running. In this case, VisibleSpeak can directly attempt to re-invoke the ASR engine's initialization interface through the voicekit to restore the voice control functionality based on VisibleSpeak.

[0212] In other embodiments, when a recording status change event is detected, VoiceKit may have already been activated by other voice functions. Therefore, in response to a recording status change event, the status of VoiceKit can be queried directly.

[0213] In the technical solution provided in this application embodiment, when voice control is restored, it can be determined whether the voicekit needs to be initialized based on the voicekit's state. If the voicekit has already started, it is not necessary to re-initialize the voicekit; instead, a call request can be sent directly to the ASR engine through the voicekit to re-initialize the ASR engine. Therefore, when the ASR engine switches between different voice control functions, the number of times the voicekit is deregistered and initialized is reduced, latency during service switching is lowered, and functional waste is minimized.

[0214] The S416.ASR engine determines whether the current engine state is idle.

[0215] In some embodiments, the ASR engine's state can be divided into the following states based on whether the ASR engine is being used: idle state, initializing state, occupied state, and deregistration state. Among them, the idle state is the state after deregistration is completed.

[0216] In some embodiments, the ASR engine can obtain the current engine state from the state queue. Further, the ASR engine can determine whether the current engine state is idle based on whether the state queue contains information related to voice control functions. For example, if the state queue contains information related to voice control functions, it indicates that the ASR engine is currently being used by that voice control function; that is, the current engine state of the ASR engine is not idle. If the state queue does not contain information related to voice control functions, it can be determined that the current ASR engine is not being used, and the current engine state is idle.

[0217] In an embodiment where the ASR engine generates the call parameters corresponding to the current user after initialization, the ASR engine can also determine whether its current state is idle by querying whether it has saved the call parameters. For example, if the ASR engine has saved the call parameters, it indicates that the ASR engine is currently being used by the voice control function; that is, the current state of the ASR engine is not idle. If the ASR engine has not saved the call parameters, it can be determined that the current ASR engine is not being used, and its current state is idle.

[0218] Furthermore, in some embodiments, when the ASR engine is in an idle state, the ASR engine's initialization interface can be invoked. And, in the idle state, in response to a call request to the initialization interface, the ASR engine can directly perform initialization.

[0219] While the ASR engine is in the initialization or deregistration state, calling the ASR engine's initialization interface is not allowed. In some embodiments, if an initialization request for the ASR engine is received while in the initialization or deregistration state, the ASR engine may ignore the request.

[0220] In other embodiments, when the ASR engine is in an occupied state, the ASR engine's initialization interface can be invoked. For example, in the occupied state, the ASR engine, in response to a call request for the initialization interface, will unregister and reload under certain conditions. Taking scenarios S401-S416 as an example, after the ASR engine initializes the visible-and-speakable service data 1, it receives a call request 2 from the voice assistant. In response to this call request 2, the ASR engine can first unregister the visible-and-speakable service data 1, and then initialize the service data to the service data 2 corresponding to the voice assistant.

[0221] As shown in S412 and S413, in response to the voice assistant's call request 3 to the ASR engine's deregistration interface, the deregistration of the voice assistant's corresponding service data 2 will begin. Therefore, in some embodiments, when the "See and Speak" call request 4 for the initialization interface is received via VoiceKit, the voice assistant may not have completed deregistration, meaning the ASR engine is in the process of deregistering. At this time, the ASR engine's initialization interface is not allowed to be called. In other words, the ASR engine cannot respond to the "See and Speak" call request 4 to initialize at this time and must wait for the voice assistant to complete its deregistration before responding.

[0222] In some embodiments, if the result of S416 is negative, the ASR engine may return to continue obtaining the current engine state and determine whether the current engine state is in an idle state.

[0223] Alternatively, in some other embodiments, when the voice assistant receives call request 4 sent via VoiceKit, the ASR engine has already released itself, and the ASR engine is in an idle state. At this time, the ASR engine allows the invocation of its initialization interface. That is, the ASR engine can directly respond to call request 4 and re-initialize, as in S417.

[0224] In some embodiments, after the ASR engine cancels the service data 2 corresponding to the voice assistant in response to call request 3, the state queue still contains relevant information about "See and Speak". Therefore, the ASR engine will not immediately unbind from the module responsible for the ASR engine in VoiceKit, but can wait for "See and Speak" to resend the call request for the initialization interface. This avoids the problem of "See and Speak" being unable to resume calling the ASR engine due to the ASR engine being unbound from VoiceKit.

[0225] S417. In response to call request 4, the ASR engine initializes the service data to the visible, speakable corresponding service data 1.

[0226] In some embodiments, after S417 described above, the ASR engine regenerates and saves the call parameters corresponding to the visible, i.e., the talkable parameters.

[0227] In the technical solution provided in this application embodiment, for a scenario where "Visible and Talkable" is started first, followed by the voice assistant, then exited, and then "Visible and Talkable" is restored, both the voice assistant and "Visible and Talkable" can directly send call requests to the ASR engine through VoiceKit while the ASR engine is already running. The ASR engine will also respond to the call requests based on its current engine state. Since VoiceKit does not need to repeatedly deregister and re-initialize during this process, the modules in VoiceKit responsible for non-ASR engines can be avoided from repeatedly unbinding and binding during the ASR engine's business switching between "Visible and Talkable" and the voice assistant. This allows for separate deregistration and reconstruction of the ASR recognition engine, retaining the shared parts in VoiceKit. This reduces the latency of the ASR engine during this business switching process, improves business switching efficiency, and reduces unnecessary functional waste.

[0228] In some cases, it's possible that after a voice control function is activated, another activation event for that same function is detected. Taking a voice assistant as an example, after detecting the wake word, the phone can launch the voice assistant, which then invokes the ASR engine. Subsequently, if the phone detects a preset action on a preset button, it may trigger the voice assistant to launch again. For the voice assistant, after it has already launched and invoked the ASR engine, detecting a preset action on a preset button may trigger a re-invocation of the ASR engine. However, when a preset action on a preset button is detected, the ASR engine may still be being used by the voice assistant.

[0229] At this point, if the voice assistant responds to the preset operation of the preset buttons and executes the ASR engine call process again, such as sending call request 5 for the initialization interface to the ASR engine via VoiceKit, then the ASR engine might respond to call request 5 by executing a deregistration and re-initialization process. However, in responding to call request 5, the ASR engine first deregisters the service data 2 corresponding to the voice assistant and then re-initializes it. Understandably, the process executed by the ASR engine in response to call request 5 is a completely unnecessary step that wastes the phone's power consumption.

[0230] In other embodiments, taking the "See and Speak" voice control function as an example, after "See and Speak" starts and invokes the ASR engine, it may be interrupted by audio / video calls or recording functions. Subsequently, when the audio / video call or recording function exits, a recording status change event will be triggered. "See and Speak" detects this recording status change event and can resume voice control based on the ASR engine. When "See and Speak" resumes using the ASR engine for voice control, the ASR engine is still in a state of being used by "See and Speak". At this time, if, in response to the recording status change event, "See and Speak" re-executes the process of invoking the ASR engine, it will also cause the ASR engine to perform unnecessary deregistration and reloading steps.

[0231] As described in the above embodiments, in some embodiments, after the ASR engine completes initialization, it also generates call parameters corresponding to the voice control function currently calling the ASR engine. Therefore, in some embodiments, to avoid the same voice control function repeatedly calling the ASR engine, resulting in wasted functionality, the ASR engine can also add a query interface. This query interface is used to support querying the current call parameters of the ASR engine. In the above example, after the voice assistant starts and detects a preset operation for a preset button, it can query the current call parameters of the ASR engine if it is determined that the ASR engine is in use. Then, it uses the current call parameters to determine whether the voice control function currently using the ASR engine is consistent with itself. If they are consistent, it means that the voice assistant is currently using the ASR engine, so the voice assistant does not need to re-execute the process of calling the ASR engine. If they are inconsistent, it means that the voice control function currently using the ASR engine is not the voice assistant, and in this case, the voice assistant needs to re-execute the process of calling the ASR engine in response to the preset operation for the preset button.

[0232] The above embodiments use a voice assistant as an example for illustration, and the same scenario applies to "what you see can be spoken". For details, please refer to [link / reference]. Figure 11 This illustrates the flow of the speech recognition engine invocation method in some other embodiments. In this embodiment, after S407 and before S408, the method further includes S501-S503. Figure 11 A detailed description of the method steps shown can be found in this application. Figure 10 The detailed description of the corresponding method steps is not repeated here.

[0233] The S501.ASR engine generates a second call parameter that corresponds to what is visible and can be said, and saves the second call parameter as the current call parameter.

[0234] The S502.ASR engine sends the second call parameter to what is visible can be said.

[0235] S503. It can be seen that the second call parameter is saved.

[0236] As can be seen, after saving the second call parameters, when voice control trigger event 1 (such as a recording status change event) is detected again, the current call parameters of the ASR engine are first queried. By comparing whether the current call parameters of the ASR engine are consistent with the second call parameters, it can be determined whether the voice control function currently using the ASR engine is "Visible and Speakable". This allows it to determine whether the ASR engine needs to be reinitialized.

[0237] It should be noted that after VisibleSpeak exits, VisibleSpeak will call the ASR engine's deregistration interface via VoiceKit to release the ASR engine. In some embodiments, VisibleSpeak will also delete the second call parameter after releasing the ASR engine. Similarly, for the ASR engine, if no voice control function is currently using the ASR engine, the currently saved call parameters of the ASR engine can be deleted.

[0238] Please continue to refer to Figure 11 In this embodiment, similarly, after S411, the ASR engine will regenerate and save the new call parameters based on the voice assistant, and return the newly generated call parameters to the voice assistant, as in S504-S506.

[0239] The S504.ASR engine generates the first call parameter corresponding to the voice assistant and changes the current call parameter to the first call parameter.

[0240] The S505.ASR engine returns the first call parameters to the voice assistant.

[0241] S506. The voice assistant saves the first call parameters.

[0242] Furthermore, in the above embodiments, after successfully invoking the ASR engine, the voice control function will receive the invocation parameters sent by the ASR engine and save these parameters. Therefore, in some embodiments, the voice control function can determine whether it has previously invoked the ASR engine by querying whether it has saved the invocation parameters. For example, it can be said that the voice control function can determine whether it has previously invoked the ASR engine by querying whether it has saved the ASR engine's invocation parameters.

[0243] In some embodiments, S409 may specifically include: in response to the second event, starting the voice assistant and querying whether the voice assistant has saved the calling parameters of the ASR engine. Further, S410 may specifically include: if the voice assistant has not saved the calling parameters from the ASR engine, sending a call request 2 for the initialization interface of the ASR engine to the ASR engine via VoiceKit.

[0244] Similarly, for the "See-It-All-Speak" feature, after detecting a recording status change event, the S415 "See-It-All-Speak" can first query whether it has saved the ASR engine's call parameters to determine whether the ASR engine has been called. Because in Figure 10 In this embodiment, prior to S415, it is evident that the second calling parameter has indeed been saved; therefore, it is evident that the current calling parameters of the ASR engine need to be queried to determine whether the ASR engine needs to be reinitialized. In this embodiment, prior to S415, the above method may further include S507-S509:

[0245] S507. If the second call parameter is stored in Visible and Sayable, in response to the recording state change event, Visible and Sayable sends a first query request to the ASR engine via Voicekit.

[0246] S508. In response to the first query request, the ASR engine sends the current call parameters to the visible-to-say engine.

[0247] S509. It can be seen that the current calling parameters are consistent with the second calling parameters.

[0248] In some embodiments, if the current invocation parameters match the second invocation parameters, then the current ASR engine is consistent with the ASR engine required by Visible and Speak. In this case, Visible and Speak does not need to reinitialize the ASR engine and can use it to begin speech recognition.

[0249] In other embodiments, if the current call parameter and the second call parameter are inconsistent, it indicates that the current ASR engine is different from the ASR engine required by VisibleSpeak. In this case, VisibleSpeak needs to reinitialize the ASR engine before it can use the ASR engine to implement voice control functions based on VisibleSpeak. In this embodiment, since the ASR engine has already changed the current call parameter to the first call parameter in S504, the judgment result of S509 is negative. That is, the second call parameter saved by VisibleSpeak is inconsistent with the current call parameter.

[0250] Furthermore, if it is known that the second call parameter is inconsistent with the current call parameter, the ASR engine's initialization interface can be called through voicekit, so that the ASR engine initializes the service data to the service data 1 corresponding to "visible and speakable".

[0251] In the technical solution provided in this application embodiment, after the ASR engine completes initialization, it returns the generated call parameters to the voice function (such as "See and Speak") currently using the ASR engine for storage. Simultaneously, the ASR engine also provides a query interface for call parameters, allowing voice functions that want to call the ASR engine to retrieve the current call parameters. For "See and Speak," which wants to call the ASR engine, the current call parameters can be compared with the previously saved call parameters. If the comparison result is consistent, it indicates that the current ASR engine is consistent with the ASR engine required by "See and Speak." Therefore, "See and Speak" can directly use the current ASR engine without re-initializing it. This avoids the redundant steps of deregistering service data 1 and re-initializing it, reducing power consumption waste.

[0252] Furthermore, if the comparison result between the current call parameters and the previously saved call parameters is inconsistent, it indicates that the current ASR engine is inconsistent with the ASR engine required by VisibleSpeak. In this case, the current ASR engine needs to be deregistered, and VisibleSpeak needs to reinitialize the ASR engine to enable voice control functionality based on VisibleSpeak. This ensures that VisibleSpeak can correctly use the ASR engine to achieve voice control.

[0253] In other embodiments, the first voice control function may be "Speak Now as You See It," and the second voice control function may be a voice assistant. The following example illustrates a scenario where "Speak Now as You See It" is activated first, followed by the voice assistant, and then the voice assistant exits and "Speak Now as You See It" is restored. In this scenario, the detailed real-time steps of the speech recognition engine's invocation method can be found above. Figure 10 and Figure 11 The detailed description is in the flowchart.

[0254] When VisibleTalk first starts, VoiceKit is in a pre-started state. In some embodiments, VisibleTalk may first determine whether it has saved call parameters from the ASR engine. Since VisibleTalk is starting for the first time, it has not saved call parameters from the ASR engine. Therefore, VisibleTalk sends a call request (such as call request 6) to VoiceKit for the initialization interface of the ASR engine, and VoiceKit needs to initialize first. Then, VoiceKit forwards call request 6 to the ASR engine.

[0255] In response to call request 6, the ASR engine initializes the service data to correspond to the service data that is visible and succinct, thus providing services to Visible and Succinct. Simultaneously, the ASR engine generates call parameters corresponding to Visible and Succinct and returns these parameters to Visible and Succinct for storage. The ASR engine saves the current call parameters as the corresponding call parameters for Visible and Succinct.

[0256] Subsequently, when the voice assistant needs to be launched, as a secondary voice control function, the voice assistant can first determine whether VoiceKit is in the launched state. Since it can be said that VoiceKit has already been launched during the launch process, if it is launched during voice startup, the voice assistant can directly send a call request for the initialization interface to the ASR engine through VoiceKit (such as call request 7).

[0257] In response to call request 7, the ASR engine will unregister the visible / speakable service data and then initialize the service data to match the voice assistant's service data. Furthermore, the ASR engine will generate call parameters corresponding to the voice assistant and return these parameters to the voice assistant for storage. Simultaneously, the ASR engine will update the current call parameters to match the voice assistant's parameters.

[0258] Subsequently, when the voice assistant exits, it will send a call request for the deregistration interface to the ASR engine via VoiceKit (e.g., call request 8) to cause the ASR engine to deregister the service data corresponding to the voice assistant. At the same time, the voice assistant triggers a recording status change event.

[0259] After detecting a recording status change event, VisibleTalk can resume voice control. VisibleTalk can first check if VoiceKit is running. As seen from the previous steps, VoiceKit is already running when VisibleTalk resumes. Therefore, VisibleTalk does not need to re-initialize VoiceKit when resuming voice control.

[0260] Furthermore, VisibleSay can determine whether it has saved the call parameters from the ASR engine. As the above steps show, after VisibleSay starts for the first time, it receives and saves the call parameters returned by the ASR engine corresponding to VisibleSay. Therefore, when VisibleSay resumes voice control, it can determine that it has saved the call parameters from the ASR engine. At this time, VisibleSay can query whether the current call parameters of the ASR engine are consistent with the corresponding call parameters of VisibleSay.

[0261] Because VisibleSay, in responding to call request 7, initializes its service to the voice assistant by updating the current call parameters to match those of the voice assistant, when VisibleSay resumes voice control, it will find that the current call parameters of the ASR engine are inconsistent with those of VisibleSay. At this point, VisibleSay needs to reinitialize the ASR engine. With VoiceKit running, VisibleSay sends a call request (such as call request 9) to the ASR engine for the initialization interface via VoiceKit. Afterwards, once the service data corresponding to the voice assistant has been deregistered, the ASR engine can respond to call request 9 and reinitialize the service data to match the service data of VisibleSay.

[0262] As described in the above embodiments, in scenarios where "Visible and Speakable" starts first, followed by the voice assistant, and then the voice assistant exits and resumes "Visible and Speakable," the switching of the ASR engine from "Visible and Speakable" to the voice assistant, and vice versa, involves the voice control function sending a call request for the initialization interface directly to the ASR engine through the voicekit while the voicekit is running. This allows the ASR engine to automatically deregister and reinitialize itself when switching services. This avoids multiple destructions and initializations of the voicekit during service switching, reducing latency and minimizing wasted functionality.

[0263] In other embodiments, the first voice control function may be "visible and speakable," and the second voice control function may be "instantaneous command." See also... Figure 12 It shows a flowchart illustrating the calling method of the speech recognition engine in a scenario where "visible and speakable" is started first, followed by "free command" and then exited by "free command" to restore "visible and speakable".

[0264] S601. Visible and speakable receive voice control trigger event 3.

[0265] In this embodiment of the application, the first voice control function is that it can be spoken as soon as it is seen; therefore, the above-mentioned voice control trigger event 3 is the first voice control trigger event.

[0266] S602. In response to voice control trigger event 3, activate the visible and speakable voice control function.

[0267] S603. It can be seen that a call request 10 is sent through the voicekit's initialization interface for the ASR engine.

[0268] It is evident that, in response to the aforementioned voice control trigger event, the system first determines whether VoiceKit is running. Figure 12 In the scenario shown, the phone did not start VoiceKit before S601. That is to say, when VoiceKit is received, it is in an inactive state. Therefore, it is to say that after VoiceKit sends a call request 10 to the ASR engine initialization interface in response to VoiceKit in response to VoiceKit trigger event 3, VoiceKit needs to be initialized first, as in S604.

[0269] S604. In response to call request 10, voicekit is initialized.

[0270] S605.voicekit sends a call request to the ASR engine for the ASR engine's initialization interface 10.

[0271] The S606.ASR engine responds to call request 10 and binds to VoiceKit.

[0272] The S607.ASR engine initializes the service data to the visible and speakable corresponding service data 3.

[0273] In this embodiment of the application, the first voice control function is visible and can be spoken; therefore, the above-mentioned service data 3 is the first service data.

[0274] S608. Mobile phone receives voice control trigger event 4.

[0275] In some embodiments, the voice control trigger event 4 may specifically be a preset scenario trigger event. In some embodiments, the voice control function of the user command will be triggered in scenarios such as the alarm clock ringing, the incoming call ringing, and answering the phone. Therefore, when scenarios such as the alarm clock ringing, the incoming call ringing, and answering the phone are detected, the voice control function of the user command will be activated.

[0276] The voice control function of the "SuiXin Command" requires the use of the ASR engine. In some embodiments, "SuiXin Command" can directly interface with the ASR engine.

[0277] S609. In response to voice control trigger event 4, activate the voice control function of the command.

[0278] S610. The Optional instruction sends a call request to the ASR engine for the ASR engine's initialization interface 11.

[0279] In response to call request 11, the S611.ASR engine unregisters the visible service data 3 and initializes the service data to the service data 4 corresponding to the command.

[0280] S612. In response to the second exit event, the 'Free Command' exits, and the 'Free Command' sends a call request 12 to the ASR engine for the ASR engine's deregistration interface.

[0281] In response to call request 12, the S613.ASR engine cancels the service data corresponding to the whimsical command 4.

[0282] S614. The recording status change event is triggered by the command.

[0283] S615. It can be seen that, in response to the recording state change event, a call request for the ASR engine's initialization interface is sent to the ASR engine via voicekit 13.

[0284] The S616.ASR engine determines whether the current engine state is idle.

[0285] S617. In response to call request 13, the ASR engine initializes the service data to the visible and speakable corresponding service data 3.

[0286] In this embodiment, the "See and Speak" function is activated first, followed by the "At Will" command, and then exited. Therefore, after exiting the "At Will" command, the voice control function based on "See and Speak" can be restored. Furthermore, since the first voice control function is "See and Speak," the aforementioned recording status change event also belongs to the first voice control trigger event.

[0287] During the S601's "See and Say" startup and ASR engine invocation process, the voicekit has already been initialized. However, during the "Follow the Command" startup process, the "Follow the Command" directly sends invocation request 11 to the ASR engine, and the ASR engine can respond to invocation request 11 by deregistering the "See and Say" service data 3 and initializing the service data to the service data 4 corresponding to the "Follow the Command". In other words, the ASR engine can automatically complete the deregistration and reload process based on invocation request 11 without deregistering the voicekit. Therefore, when "See and Say" re-invokes the ASR engine in response to a recording status change event, the voicekit is still running. Thus, "See and Say" can directly send invocation request 13 to the ASR engine through the voicekit. Therefore, during the ASR engine's switching between different services, the problem of repeated voicekit deregistration and reinitialization can be avoided, reducing latency during service switching and minimizing power consumption waste.

[0288] in addition, Figure 13 The overall flow of the calling method in the aforementioned speech recognition engine is shown. It should be noted that... Figure 13The APP1 and APP2 shown can correspond to any voice control function that needs to call the ASR engine to implement the corresponding function. For example, the voice control function can be something that can be seen and spoken, a voice assistant, or a command, etc. In this embodiment, the method of calling the voice recognition engine includes steps 1-5:

[0289] Step 1: APP2 queries the application status queue in the ASR engine by caching the ASR engine handle and the corresponding ID. If the current application status queue contains a handler corresponding to APP1, it means that the current user of the ASR engine is APP2, and there is no need to switch or initialize. At this time, no operation needs to be performed.

[0290] If the current application status queue does not contain information related to APP2, it is assumed that the current ASR engine user is not APP2. In this case, the application status queue is used to check whether application APP1 is in the set switchable whitelist. If APP1 is not in the switchable whitelist, a switching failure is returned to APP2, indicating that APP2 does not have permission to switch the use of the ASR engine. If APP1 is in the switchable whitelist, proceed to step 2.

[0291] Step 2: Check if the engine status used by APP1, which is currently occupying the ASR engine, is in the process of being deregistered. If it is in the process of being deregistered, proceed to Step 3; otherwise, proceed to Step 4.

[0292] Step 3: After APP1 finishes deregistering from the ASR engine, delete APP1's relevant information from the application status queue. Then, insert the application information of APP2, which currently needs to use the ASR engine, into the head of the application status queue. At the same time, the ASR engine initializes its service data to the service data corresponding to APP2 and begins normal use. You can then optionally proceed to Step 5.

[0293] Step 4: Unregister the ASR engine resources currently being used by APP1, and reinitialize the service data to the ASR engine for use by APP2. Simultaneously, insert the relevant information of APP2 into the head of the current application state queue, and generate and store the handler corresponding to APP2. Then, optionally proceed to Step 6.

[0294] Step 5: When APP2 calls the ASR engine's deregistration interface, the engine resources used by APP2 are deregistered, ending APP2's call to the ASR engine.

[0295] Step 6: When APP2 calls the ASR engine cancellation interface, it cancels the relevant engine resources and re-initializes the ASR engine. At the same time, it deletes the relevant status information of APP2 from the head of the application status queue and restores the status based on the relevant status information of APP2 cached in the status queue, so that the suspended APP1 can continue to use it.

[0296] As can be seen from the description of the above embodiments, in Figure 13 In the illustrated process, step 4 involves deregistering the engine resources used by APP1 and re-initializing the ASR engine for use by APP2, effectively suspending APP1. In some embodiments, while APP1 is suspended and APP2 is using the ASR engine, the suspended APP1 may call ASR engine-related interfaces for some reason (such as during testing or process termination).

[0297] In this scenario, if APP1 calls the ASR engine's destroy interface, the suspended state information processing for APP1 in the application state queue is removed to ensure the normal operation of the ASR engine. Therefore, when APP2 ends its call to the ASR engine in step 6 above, the phone will no longer resume APP1's call to the ASR engine based on the relevant information of APP1 in the application state queue. This ensures the normal operation of the ASR engine.

[0298] If the suspended APP1 calls a non-destroy interface of the ASR engine, the ASR engine can ignore it. This ensures that APP2 can use the ASR engine to implement the corresponding voice control functions normally.

[0299] Other embodiments of this application provide an electronic device, which may be the aforementioned mobile phone. The electronic device may include a microphone, a memory, and one or more processors. The recording device and the memory are coupled to the processors. The recording device is used to acquire audio data. The memory is also used to store computer program code, which includes computer instructions. When the processor executes the computer instructions, the electronic device can perform various functions or steps performed by the mobile phone in the above method embodiments. In some embodiments, the electronic device may further include a display screen for displaying the interface of the electronic device, and the displayed interface may also display prompt icons and other information. The structure of the electronic device can be referred to... Figure 7 The structure of the electronic device 100 shown.

[0300] This application also provides a chip system, such as... Figure 14As shown, the chip system 140 includes at least one processor 1401 and at least one interface circuit 1402. The processor 1401 and the interface circuit 1402 are interconnected via lines. For example, the interface circuit 1402 can be used to receive signals from other devices (e.g., the memory of an electronic device). As another example, the interface circuit 1402 can be used to send signals to other devices (e.g., the processor 1401). Exemplarily, the interface circuit 1402 can read instructions stored in memory and send those instructions to the processor 1401. When the instructions are executed by the processor 1401, the electronic device can perform the steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in this application embodiment.

[0301] This application also provides a computer-readable storage medium including computer instructions that, when executed on the aforementioned electronic device (such as a mobile phone), cause the electronic device to perform various functions or steps performed by the mobile phone in the above method embodiments.

[0302] This application also provides a computer program product that, when run on a computer, causes the computer to perform the various functions or steps performed by the mobile phone in the above method embodiments. The computer can be an electronic device, such as a mobile phone.

[0303] Through the above description of the embodiments, those skilled in the art can clearly understand that, for the sake of convenience and brevity, only the division of the above functional modules is used as an example. In actual applications, the above functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0304] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of modules or units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another apparatus, or some features may be ignored or not executed. Furthermore, the mutual coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0305] The units described as separate components may or may not be physically separate. A component shown as a unit can be one or more physical units; that is, it can be located in one place or distributed in multiple different locations. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0306] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0307] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solutions of the embodiments of this application, in essence, or the parts that contribute to the prior art, or all or part of the technical solutions, can be embodied in the form of a software product. This software product is stored in a storage medium and includes several instructions to cause a device (which may be a microcontroller, chip, etc.) or processor to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0308] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be included within the scope of protection of this application. Therefore, the scope of protection of this application should be determined by the scope of the claims.

Claims

1. A method for invoking a speech recognition engine, characterized in that, The method is applied to an electronic device having a first voice control function; the electronic device includes a speech recognition (ASR) engine and a voice kit, the voice kit being a voice suite used to invoke the ASR engine; The first voice control module corresponding to the first voice control function interfaces with the ASR engine through the voicekit; the method includes: The first voice control module receives the first voice control trigger event; When the voicekit is in the startup state, the first voice control module responds to the first voice control trigger event and sends a first call request through the voicekit's initialization interface for the ASR engine; If the current user of the ASR engine is not the first voice control module, and the ASR engine meets the preset switching conditions, the ASR engine responds to the first call request and initializes the service data of the ASR engine to the first service data corresponding to the first voice control module, so as to provide services for the first voice control function; the preset switching conditions are used to indicate that the ASR engine allows the current user to switch to the first voice control module.

2. The method according to claim 1, characterized in that, When the voicekit is in the started state, the first voice control module responds to the first voice control trigger event by sending a first call request through the voicekit's initialization interface for the ASR engine, including: If the current user of the ASR engine is not the first voice control module, then when the voicekit is in the startup state, the first voice control module responds to the first voice control trigger event and sends a first call request through the voicekit's initialization interface for the ASR engine.

3. The method according to claim 2, characterized in that, After the ASR engine responds to the first call request and initializes its service data to the first service data corresponding to the first voice control module, the method further includes: The ASR engine generates a first call parameter corresponding to the first voice control module, and saves the first call parameter as the current call parameter of the ASR engine; The ASR engine sends the first call parameter to the first voice control module; The first voice control module receives and saves the first call parameters.

4. The method according to claim 3, characterized in that, The current user of the ASR engine is not the first voice control module, including: When the first voice control trigger event is received, the first voice control module does not save the call parameters from the ASR engine; The first voice control trigger event includes the first start event of the first voice control function.

5. The method according to claim 4, characterized in that, In response to the first call request, the ASR engine initializes its service data to the first service data corresponding to the first voice control module, including: In response to the first call request, the ASR engine cancels the service data corresponding to the current user; The ASR engine initializes its service data to the first service data corresponding to the first voice control module.

6. The method according to claim 3, characterized in that, The current user of the ASR engine is not the first voice control module, including: When the first voice control trigger event is received, the first voice control module stores the call parameters from the ASR engine, and the call parameters stored by the first voice control module are inconsistent with the current call parameters of the ASR engine. The first voice control trigger event includes: a recording status change event triggered when the current user of the ASR engine exits.

7. The method according to claim 6, characterized in that, The electronic device also has a second voice control function; the current user of the ASR engine is the second voice control module corresponding to the second voice control function, and the current call parameters of the ASR engine are the second call parameters corresponding to the second voice control function. Before the first voice control module receives the first voice control trigger event, the method further includes: The second voice control module receives the second start event of the second voice control function; In response to the second startup event, the second voice control module sends a second call request through the voicekit's initialization interface for the ASR engine; In response to the second call request, the service data of the ASR engine is initialized to the second service data corresponding to the second voice control function; The second voice control module receives the exit event of the second voice control function; In response to the exit event, the second voice control module triggers a recording status change event and sends a third call request to the ASR engine's deregistration interface via the voicekit. In response to the third call request, the ASR engine cancels the second service data; Specifically, saving the first call parameter as the current call parameter of the ASR engine includes: the ASR engine updating the current call parameter from the second call parameter to the first call parameter.

8. The method according to any one of claims 2-7, characterized in that, The method further includes: If the current user of the ASR engine is the first voice control module, then the first voice control module uses the ASR engine for voice control through the voicekit.

9. The method according to any one of claims 1-7, characterized in that, After the first voice control module receives the first voice control trigger event, the method further includes: When the voicekit is not started, the first voice control module responds to the first voice control trigger event and sends a first call request to the voicekit; The voicekit is activated in response to the first call request; The voicekit sends the first call request to the ASR engine; In response to the first call request, the ASR engine initializes its service data to the first service data corresponding to the first voice control module.

10. An electronic device, characterized in that, The electronic device includes: a processor, a memory, and a microphone; the memory and the microphone are respectively coupled to the processor. The microphone is used to collect audio data; the memory stores computer program code, which includes computer instructions that, when executed by the processor, cause the electronic device to perform the method as described in any one of claims 1-9.

11. A computer-readable storage medium, characterized in that, Includes computer instructions that, when executed on an electronic device, cause the electronic device to perform the method as described in any one of claims 1-9.

Images