Voice recognition engine calling method and electronic equipment

Abstract

The invention provides a voice recognition engine calling method and electronic equipment, relates to the technical field of electronics, and is used for reducing time delay during switching of an ASR engine and reducing unnecessary power consumption. The method is applied to the electronic equipment, and the electronic equipment has a first voice control function. The electronic equipment comprises a voice recognition ASR engine and a voice suite voicekit. A first voice control module corresponding to the first voice control function is in butt joint with the ASR engine through voicekit; the method comprises the steps that a first voice control module receives a first voice control trigger event; under the condition that the voicekit is in a starting state, the first voice control module responds to a first voice control trigger event, and sends a first calling request for an initialization interface of the ASR engine through the voicekit; and the ASR engine initializes service data of the ASR engine into first service data corresponding to the first voice control module in response to the first calling request so as to provide services for the first voice control function.

Description

Technical Field

[0001] The embodiments of the present application relate to the field of electronic technology, and more particularly to a method for calling a speech recognition engine and an electronic device. Background Art

[0002] With the development of technology, it is now possible for users to control electronic devices through voice. In the same electronic device, there may be more than two different voice control functions. Different voice control functions can be used in different scenarios to achieve different voice control effects. The voice control function usually requires the use of an automatic speech recognition (ASR) engine to recognize the collected user voice.

[0003] Different voice control functions can share the same ASR engine, and some voice control functions need to be connected to the ASR engine through the same voice kit (voicekit). In the related art, the ASR engine is a single instance in the electronic device, and only one service access is allowed at the same time, that is, the ASR engine can only provide services for one voice control function at a time. At present, in the scenario where electronic devices switch to use different voice control functions one after another, there is a problem of long delay when switching services, which easily causes unnecessary power consumption. Summary of the invention

[0004] The embodiments of the present application provide a method for calling a speech recognition engine and an electronic device, which are used to reduce the delay when switching an ASR engine and reduce unnecessary power consumption.

[0005] To achieve the above objectives, the embodiments of the present application adopt the following technical solutions:

[0006] In a first aspect, a method for calling a speech recognition engine is provided, the method being applied to an electronic device, the electronic device having a first speech control function. The electronic device comprises a speech recognition ASR engine and a speech kit voicekit; a first speech control module corresponding to the first speech control function is connected to the ASR engine through the voicekit; the method comprises:

[0007] The first voice control module receives the first voice control trigger event. When voicekit is in the startup state, the first voice control module responds to the first voice control trigger event and sends a first call request to the initialization interface of the ASR engine through voicekit. In this way, the step of deregistering voicekit and then reinitializing when the first voice control module calls the ASR engine is avoided. Instead, the ASR engine responds to the first call request and initializes the service data of the ASR engine to the first service data corresponding to the first voice control module, so that the ASR engine provides services for the first voice control function.

[0008] In this solution, when the voice control function needs to call the ASR engine through voicekit, the status of voicekit is first determined. If voicekit is already in the startup state, the first voice control module can no longer deregister and initialize voicekit. Instead, the initialization interface of the ASR engine is directly called through voicekit to initialize voicekit and provide services for the first voice control function. In this way, in a scenario where two or more voice control functions using the same ASR engine are started one after another, repeated deregistration and initialization of voicekit can be avoided, thereby avoiding repeated unbinding and binding between other modules in voicekit other than the ASR engine and other engines. This reduces the delay when the ASR engine switches services and reduces unnecessary power consumption.

[0009] In a possible implementation of the first aspect, when voicekit is in the startup state, before the first voice control module responds to the first voice control trigger event and sends the first call request to the initialization interface of the ASR engine through voicekit, the first voice control module may first determine whether the current user of the ASR engine is the first voice control module. If the current user of the ASR engine is not the first voice control module, when voicekit is in the startup state, the first voice control module responds to the first voice control trigger event and sends the first call request to the initialization interface of the ASR engine through voicekit.

[0010] In this solution, if the current user of the ASR engine is not the first voice control module itself, it means that the current ASR engine cannot be used directly by the first voice control module, but needs to be reinitialized. Therefore, after determining that the current user of the ASR engine is not the first voice control module, the first voice control module will send a call request of the initialization interface to the ASR engine through voicekit to initialize the ASR engine.

[0011] In a possible implementation of the first aspect, after the ASR engine responds to the initial completion of the call request of the voice control module, it will generate a call parameter corresponding to the current user and save the call parameter corresponding to the current user as the current call parameter. Further, the ASR engine can also return the current call parameter to the current user for storage. However, if the current user of the ASR engine changes, the ASR engine will update the current call parameter accordingly.

[0012] Therefore, if the first voice control module has called the ASR engine and has not actively exited the call to the ASR engine, the first voice control module will save the call parameters from the ASR engine. The call parameters correspond to the first voice control module. When the first voice control trigger event is received, if it is detected that the first voice control module has saved the call parameters of the ASR engine, the first voice control module can first determine whether the current call parameters of the ASR engine are consistent with the call parameters saved by itself, so as to determine whether the current call parameters of the ASR engine are the first voice control module.

[0013] In a possible implementation of the first aspect, when the first voice control module saves the calling parameters of the ASR engine, but the calling parameters are inconsistent with the current calling parameters of the ASR engine, it can be determined that the current user of the ASR engine is not the first voice control module.

[0014] Furthermore, if the first voice control module saves the calling parameters of the ASR engine, it means that the first voice control module has called the ASR engine. If the calling parameters of the first voice control module are inconsistent with the current calling parameters of the ASR engine, it means that the ASR engine was preempted by other voice control functions during the use of the ASR engine by the first voice control module. At this time, the first voice control module can resume the use of the ASR engine after the current user of the ASR engine exits the use of the ASR engine. Therefore, the first voice control trigger event includes: a recording status change event triggered when the current user of the ASR engine exits.

[0015] In another possible implementation of the first aspect, if the first voice control module does not save the calling parameters of the ASR engine, it means that the first voice control module has not called the ASR engine, and the current user of the ASR engine is not the first voice control module. Further, the first voice control module has not called the ASR engine, which means that the first voice control trigger event can specifically be a start event for the first voice control function.

[0016] In a possible implementation of the first aspect, if the current user of the ASR engine is the first voice control module, it means that the current ASR engine can be used by the first voice control module. At this time, the first voice control module does not need to reinitialize the ASR engine, but can directly use the ASR engine for voice control through voicekit.

[0017] In a possible implementation of the first aspect, when the first voice control module saves the calling parameters from the ASR engine and the calling parameters are consistent with the current calling parameters of the ASR engine, it indicates that the current user of the ASR engine is the first voice control module.

[0018] In this solution, before the first voice control module initiates a call request to the ASR engine in response to the first voice control trigger event, it first determines whether the current user of the ASR engine is the first voice control module. In this way, it can be avoided that when the current user of the ASR engine is the first voice control module, the first voice control module repeatedly initializes the ASR engine through the voicekit, thereby reducing unnecessary power consumption.

[0019] In a possible implementation of the first aspect, if the first voice control module does not save the call parameters from the ASR engine and detects a startup event for the first voice control module, the first voice control module needs to call the ASR engine. The ASR engine needs to switch from the current user to the first voice control module. Therefore, the above-mentioned ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module in response to the first call request, which may specifically include: the ASR engine responds to the first call request and first cancels the service data corresponding to the current user. Then, the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module. In this way, the ASR engine directly cancels and reinitializes according to the call request, without the need for voicekit to cancel and initialize. Thereby avoiding the delay caused by unnecessary cancellation and initialization of voicekit when the ASR engine switches services, and can also reduce power consumption and improve the efficiency of service switching.

[0020] In a possible implementation of the first aspect, when the call parameters saved by the first voice control module are inconsistent with the current call parameters of the ASR engine, the current user of the ASR engine may be a second voice control module corresponding to the second voice control function of the electronic device. Correspondingly, the current call parameters of the ASR engine are second call parameters corresponding to the second voice control function.

[0021] In this scheme, before the first voice control module receives the first voice control trigger event, the method also includes: the second voice control module receives the second start event of the second voice control function; the second voice control module responds to the second start event and sends a second call request to the initialization interface of the ASR engine through voicekit; responds to the second call request, initializes the service data of the ASR engine to the second service data corresponding to the second voice control function; the second voice control module receives the exit event of the second voice control function; the second voice control module responds to the exit event, triggers the recording state change event, and sends a third call request to the deregistration interface of the ASR engine through voicekit; the ASR engine responds to the third call request and deregisters the second service data. Wherein, saving the first call parameter as the current call parameter of the ASR engine includes: the ASR engine updates the current call parameter from the second call parameter to the first call parameter.

[0022] In this scheme, before the first voice control module receives the first voice control trigger event, the first voice control module first calls the ASR engine, then the second voice control module starts and calls the ASR engine, and then the second voice control module exits the use of the ASR engine. Afterwards, the first voice control module can re-request to call the ASR engine to restore the first voice control function after detecting that the second voice control module exits the use of the ASR engine. The first call request initiated by the first voice control module is initiated after the second voice control module exits the use of the ASR engine. For the ASR engine, it is necessary to first deregister the second service data corresponding to the second voice control module according to the third call request for the deregistration interface initiated by the second voice control module, and then initialize the service data of the ASR engine to the first service data corresponding to the first voice control module in response to the first call request. Thereby, the effect of restoring the first voice control function is achieved.

[0023] In this way, when the second voice control module quits using the ASR engine and resumes using the ASR engine by the first voice control module, the first voice control module can also directly send an initialization call request to the ASR engine through the voicekit when the voicekit is in the startup state. This reduces the steps of voicekit deregistration and initialization during the service switching process of the ASR engine, reduces latency, and reduces power consumption.

[0024] In a possible implementation of the first aspect, when the current user of the ASR engine is not the first voice control module, the first voice control module sends an initialization request (first call request) to the ASR engine through voicekit. After receiving the first call request, the ASR engine may also determine whether the current situation meets the preset switching condition. If so, the current user is allowed to switch to the first voice control module. The preset switching condition is used to indicate that the ASR engine allows the current user (not the first voice control module) to switch to the first voice control module. In this way, the normal use of the ASR engine by the current user can be guaranteed.

[0025] In a possible implementation of the first aspect, if voicekit is in an unstarted state when the first voice control module receives the first voice control trigger event, the first voice control module first sends a first call request to voicekit in response to the first voice control trigger event. Voicekit is first started in response to the first call request. Then, voicekit sends the first call request to the ASR engine; and the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module in response to the first call request.

[0026] In a second aspect, an electronic device is provided, comprising: a processor, a memory and a microphone; the memory and the microphone are coupled to the processor respectively. The memory is used to store computer-executable instructions, and when the electronic device is running, the processor executes the computer-executable instructions stored in the memory, so that the electronic device executes the calling method of the speech recognition engine as described in any one of the above-mentioned first aspects.

[0027] In a third aspect, a computer-readable storage medium is provided, wherein instructions are stored in the computer-readable storage medium, and when the computer-readable storage medium is executed on a computer, the computer can execute the method for calling a speech recognition engine according to any one of the first aspects.

[0028] In a fourth aspect, a computer program product comprising instructions is provided, which, when executed on an electronic device, enables the electronic device to execute the method for calling a speech recognition engine according to any one of the first aspects.

[0029] In a fifth aspect, a device (for example, the device may be a chip system) is provided, the device including a processor for supporting an electronic device to implement the functions involved in the first aspect above. In one possible design, the device also includes a memory for storing program instructions and data necessary for the electronic device. When the device is a chip system, it may be composed of a chip, or may include a chip and other discrete devices.

[0030] Among them, the technical effects brought about by any design method in the second to fifth aspects can refer to the technical effects brought about by different design methods in the first aspect, and will not be repeated here. BRIEF DESCRIPTION OF THE DRAWINGS

[0031] Figure 1 A schematic diagram of a scenario of a method for voice controlling a terminal device provided in an embodiment of the present application;

[0032] Figure 2 A schematic diagram of a scenario of a method for voice controlling a terminal device provided in an embodiment of the present application;

[0033] Figure 3 A schematic diagram of a scenario for setting a preset switch for a voice control function provided in an embodiment of the present application;

[0034] Figure 4 A schematic diagram of an interface for controlling a mobile phone to perform operations by voice provided in an embodiment of the present application;

[0035] Figure 5 A schematic diagram of an interface for controlling a mobile phone to perform operations by voice provided in an embodiment of the present application;

[0036] Figure 6 A flowchart of a method for calling a speech recognition engine in a related technology provided in an embodiment of the present application;

[0037] Figure 7 A schematic diagram of the hardware structure of an electronic device provided in an embodiment of the present application;

[0038] Figure 8 A software architecture diagram of an electronic device provided in an embodiment of the present application;

[0039] Fig. 9 A schematic diagram of the interaction of various modules when a mobile phone is controlled by voice according to an embodiment of the present application;

[0040] Fig.10 A flowchart of a method for calling a speech recognition engine provided in an embodiment of the present application;

[0041] Fig.11 A flowchart of a method for calling a speech recognition engine provided in an embodiment of the present application;

[0042] Fig.12 A flowchart of a method for calling a speech recognition engine provided in an embodiment of the present application;

[0043] Fig.13 A flowchart of a method for calling a speech recognition engine provided in an embodiment of the present application;

[0044] Fig.14A schematic diagram of the structure of a chip system provided in an embodiment of the present application. DETAILED DESCRIPTION

[0045] The following is a brief introduction to the technical terms that may be involved in the embodiments of the present application.

[0046] Voice control function:

[0047] Many electronic devices support voice control functions. Electronic devices collect user voices through microphones, analyze and recognize user voices, and execute commands corresponding to user voices, so that users can control electronic devices through voice.

[0048] Generally speaking, in order to save power consumption of electronic devices and avoid false triggering, the voice control function needs to be turned on before it can be used. For example, the user inputs a preset word (called a wake-up word) to the electronic device through voice to wake up the electronic device. After the electronic device is awakened, it can execute the instructions corresponding to the user's voice, that is, the voice control function is turned on. For example, the user can turn on or off the voice control function by turning on or off the preset switch in the human-computer interaction interface of the electronic device.

[0049] In different electronic devices, the voice control function may have different names, such as "voice control", "intelligent voice", "voice assistant", "see and speak", "voice command", "free command", "intelligent AI", etc. The specific implementation of voice control functions with different names may also be different.

[0050] Several different implementations of the voice control function are described below by way of example.

[0051] Voice Assistant:

[0052] Before a user uses a voice assistant to control an electronic device, the electronic device needs to be awakened first. In one example, the electronic device is awakened by detecting a wake-up word input by the user into the electronic device. In another example, the electronic device is awakened by detecting a user long pressing the power button. In another example, the electronic device is awakened by detecting the breath generated when the user inputs voice into the electronic device.

[0053] Generally speaking, before an electronic device is awakened, its microphone works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. The voice collected by the microphone is only detected as a wake-up word at the kernel layer, and the corresponding recording channel of the voice assistant is not started in the system and driver of the electronic device.

[0054] In response to user operations (such as receiving a wake-up word input by the user's voice), the electronic device is awakened, and the corresponding recording channel of the voice assistant is started in the system and driver. After the electronic device is awakened, the voice (audio stream) collected by the microphone is sent to the voice assistant application for processing through the recording channel corresponding to the voice assistant. In this way, the electronic device can execute the instructions corresponding to the user's voice, enabling the user to control the electronic device through voice; it can also realize functions such as dialogue with the user.

[0055] For example, Figure 1 The figure shows a schematic diagram of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 1 As shown, the mobile phone displays the desktop interface, and the user inputs the voice "Hello YOYO" into the mobile phone. In response to receiving the wake-up word "Hello YOYO", the mobile phone is awakened. For example, after the mobile phone is awakened, the voice "I am here" is played to remind the user that the mobile phone has been awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, as Figure 1 As shown, the user inputs the voice "open calendar" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the instruction corresponding to the voice "open calendar". Exemplarily, the mobile phone starts the calendar application in response to receiving the user's voice "open calendar".

[0056] For example, Figure 2 FIG. 1 shows another example of a scenario in which a user uses a voice assistant to control a mobile phone. Figure 2 As shown, the mobile phone displays the desktop interface, and the user long presses the power button. In response to receiving the user's long press of the power button operation, the mobile phone is awakened. After the mobile phone is awakened, the user can control the mobile phone by voice. For example, Figure 2 As shown, the user inputs the voice "open calendar" to the mobile phone, and the mobile phone analyzes and recognizes the user's voice and executes the instruction corresponding to the voice "open calendar". Exemplarily, the mobile phone starts the calendar application in response to receiving the user's voice "open calendar".

[0057] It is understandable that in other instances, the mobile phone can also respond to other operations of the user on the power button (such as double-clicking, etc.) to start the voice assistant.

[0058] Alternatively, in some other embodiments, the mobile phone may also display an icon corresponding to the voice assistant on the interface. When an operation on the icon corresponding to the voice assistant is detected, the mobile phone may start the voice assistant.

[0059] In some implementations, after the electronic device is awakened, the user can issue a command to the electronic device by inputting voice into the electronic device, and the electronic device executes the command corresponding to the voice. After the electronic device executes a command, or if the electronic device does not receive a command from the user through voice within a period of time (for example, within 8 seconds) after being awakened, the electronic device no longer responds to the command issued by the user through voice. For example, the electronic device will turn off the recording channel corresponding to the voice assistant. The user needs to input the wake-up word into the electronic device again to wake up the electronic device before he can issue a command to the electronic device by inputting voice into the electronic device again. In other words, after the electronic device is awakened, it enters a "short reception" state and can respond to the user's voice command within a short period of time (for example, within 8 seconds).

[0060] In some implementations, the electronic device is in a network connection state and supports entering a continuous conversation scenario after being awakened, so that the user can have a continuous conversation with the electronic device. After each broadcast, the electronic device will continue to pick up the sound without repeated awakening. Until the user exits the continuous conversation through instructions such as "exit", or the user clicks the return control / return button / exit control / exit button, or the user performs a preset return gesture on the display screen to control the electronic device to exit the continuous conversation.

[0061] It can be seen that:

[0062] It can be seen that it is implemented locally by electronic devices and does not require a network connection.

[0063] Visible and Talkable is controlled by a preset switch. Users can turn on Visible and Talkable by turning on the preset switch, and can exit Visible and Talkable by turning off the preset switch.

[0064] Taking the electronic device as a mobile phone as an example, for example, Figure 3 As shown, the user can open the setting function of the mobile phone; for example, the user clicks the application icon of the "settings" application on the desktop. In response to the user's click operation on the application icon of the "settings" application, the mobile phone displays the "settings" interface 101. The "settings" interface 101 includes a "smart voice" option 102, and the "smart voice" option 102 is used to set the smart voice function. Exemplarily, in response to the user's click operation on the "smart voice" option 102, the mobile phone displays the "smart voice" interface 103, and the "smart voice" interface 103 includes a "see and say" option 104. The user can click the "see and say" option 104 to set options related to the see and say function.

[0065] Exemplary, reference Figure 3In response to the user's click operation on the "See and Say" option 104, the mobile phone displays the "See and Say" interface 105. Optionally, the "See and Say" interface 105 includes a prompt message 106 for prompting the user how to use the See and Say function. The "See and Say" interface 105 also includes a "See and Say" switch 107. The user can click the "See and Say" switch 107 to turn on or off the "See and Say" switch. In one example, in response to receiving the user's click operation on the "See and Say" switch 107, the "See and Say" switch of the mobile phone is turned on to turn on the See and Say function. Optionally, the "See and Say" interface 105 displays a prompt message 108 for prompting the user that the See and Say function has been successfully turned on.

[0066] In one embodiment, after the see-and-speak function is turned on, the mobile phone displays a first prompt icon, and the first prompt icon indicates that the see-and-speak function has been turned on. Figure 3 As shown, after the "see and speak" switch 107 is turned on, the status bar of the mobile phone display interface displays a prompt icon 10a, indicating that the see and speak function has been turned on.

[0067] In one scenario, the preset switch corresponding to Visible and Talking on the electronic device is not turned on, and the microphone of the electronic device is not turned on. When the preset switch corresponding to Visible and Talking is turned on, the electronic device starts the microphone and starts the corresponding recording channel of Visible and Talking in the system and driver. In this way, the voice (audio stream) collected by the microphone can be sent to the Visible and Talking application for processing through the recording channel corresponding to Visible and Talking, so that the user can control the electronic device through voice.

[0068] In another scenario, the preset switch corresponding to the visible and speaking function of the electronic device is not turned on, and the microphone of the electronic device works in a power-saving mode (such as searching for signals at a lower power) to pick up sounds from the surrounding environment. When the preset switch corresponding to the visible and speaking function is turned on, the corresponding recording channel of the visible and speaking function is started in the system and driver of the electronic device. In this way, the voice (audio stream) collected by the microphone can be sent to the visible and speaking application for processing through the recording channel corresponding to the visible and speaking function, so that the user can control the electronic device through voice.

[0069] After Visible is Talking is turned on, the corresponding recording channel of Visible is Talking is started in the system and driver of the electronic device, and the electronic device enters a "long reception" state to continuously collect surrounding sounds. Users can give commands to the electronic device at any time through voice without having to enter a wake-up word to wake up the electronic device.

[0070] In one implementation, when any function on the electronic device turns on the voice recording function of the electronic device (turns on the microphone and establishes a recording channel), the electronic device will send a prompt message to the user to remind the user that the electronic device is in a voice collection state. In this way, the user's privacy can be prevented from being leaked. For example, after the visible and audible function is turned on, the electronic device enters a state of continuous voice collection, and a second prompt icon is displayed on the display interface of the electronic device. The second prompt icon indicates that the recording channel is turned on, which is used to remind the user that the microphone is collecting voice. For example, Figure 3 As shown, the status bar of the mobile phone display interface displays a prompt icon 10b, indicating that the recording channel is turned on, which is used to prompt the user that the microphone is collecting voice.

[0071] After the See and Speak function is turned on, the electronic device establishes a recording channel and continuously collects the user's voice through the microphone. The user can input voice into the electronic device at any time. The electronic device analyzes and recognizes the voice input by the user and executes the instructions corresponding to the user's voice.

[0072] In one example, if Figure 4 As shown, the mobile phone displays a desktop interface 120. The user inputs a voice message "Open calendar" to the mobile phone. In response to receiving the voice message "Open calendar" input by the user, the mobile phone starts the calendar application and displays a user interface 121 of the calendar application.

[0073] In another example, Figure 5 As shown, the mobile phone displays the interface 123 of the short video application. The user inputs the voice "pause" into the mobile phone. In response to receiving the voice "pause" input by the user, the mobile phone executes the command corresponding to the voice "pause" and displays the interface 124 of the short video application. The "play" button 125 is displayed in the interface 124 to prompt the user that the short video has been paused. The user can click the "play" button 125 to continue playing the short video. The user can also input the voice "play" into the mobile phone to continue playing the short video.

[0074] VoiceKit: It can be set in the same application as the voice control function. Some voice control functions can be connected to the ASR engine through VoiceKit. For example, Visible and Speak and Voice Assistant can be connected to the ASR engine through VoiceKit respectively.

[0075] The ASR engine can convert voice signals into text or commands based on machine recognition and understanding.

[0076] Handle: It is a calling parameter of the ASR engine, which can be used to characterize the voice control function currently using the ASR engine. After the ASR engine is called by the voice control function and initialized, it can generate a corresponding handler based on the voice control function that initiated the call request. When different voice control functions call the ASR engine, the ASR engine can generate different handlers accordingly. When the same voice control function calls the ASR engine, the ASR engine can generate the same handler.

[0077] Natural Language Understanding (NLU) is a general term for all methods, models or tasks that support machines to understand text content.

[0078] Dialogue Management (DM) is used to control the process of human-computer dialogue and decide the response to the user at the moment based on the dialogue history information.

[0079] It is now possible for users to control electronic devices through voice. For example, in the scene where the user is cooking in the kitchen and watching videos, or eating and watching dramas, the user's hands may be occupied. At this time, the user usually controls the electronic device to perform the corresponding operation by speaking voice commands. The user can control the phone to perform the corresponding operation by speaking voice commands within a certain range close to the phone. In the same electronic device, there may be more than two different voice control functions. Different voice control functions can be used in different scenarios to control electronic devices to perform different operations and achieve different control effects.

[0080] In some electronic devices, different voice control functions can share the same ASR engine. In related technologies, the ASR engine in electronic devices is usually a single instance, that is, only one service can be accessed and used at the same time, and concurrency is not supported. In addition, the voice commands supported by different voice control functions, as well as the context and hot words during voice recognition, may be different. This also shows that two different voice control functions cannot use the same ASR engine at the same time.

[0081] Therefore, when voice control function 1 uses the ASR engine, if voice control function 2 requests to call the ASR engine, usually when certain conditions are met, the ASR engine needs to switch from providing services for voice control function 1 to providing services for voice control function 2.

[0082] Take voice control function 1 as Visible and Talk, and voice control function 2 as Voice Assistant as an example. From the above description, we can see that both Visible and Talk and Voice Assistant can be connected to the ASR engine through VoiceKit. Figure 6 In the process shown, the mobile phone first starts Visible and Speak in response to the trigger operation for Visible and Speak, and Visible and Speak calls the initialization-related interface in the ASR engine through VoiceKit, so that the ASR engine initializes the service data to the service data corresponding to Visible and Speak, and provides services for Visible and Speak. In some embodiments, the ASR engine initializes the service data to the service data corresponding to Visible and Speak, which can also be called the ASR engine establishing a corresponding relationship with Visible and Speak.

[0083] Afterwards, the mobile phone can start the voice assistant in response to the trigger operation of the voice assistant. Since the ASR engine has been occupied by Visible and Speakable, the voice assistant needs to first notify Visible and Speakable to deregister the ASR engine. Specifically, Visible and Speakable calls the deregistration-related interface in the ASR engine through VoiceKit to deregister the corresponding service data of Visible and Speakable, so as to achieve the effect of Visible and Speakable releasing the ASR engine. In the process of Visible and Speakable deregistering the corresponding service data of Visible and Speakable through VoiceKit, VoiceKit will also be deregistered accordingly.

[0084] It can be seen that after releasing the ASR engine, the voice assistant can call the initialization-related interface in the ASR engine through voicekit to initialize the service data of the ASR engine to the service data corresponding to the voice assistant. Then, the voice assistant can exit in response to the exit operation for the voice assistant. At this time, the voice assistant will call the logout interface of the ASR engine through voicekit to achieve the effect of the voice assistant releasing the ASR engine.

[0085] Since Visible is still in the started state when the voice assistant exits, after the voice assistant exits, the mobile phone needs to assign the ASR engine to Visible again to restore the voice control function based on Visible. When the voice assistant exits, the recording will be exited, resulting in a change in the recording state. Therefore, in some embodiments, when the voice assistant exits, Visible can perceive the exit of the voice assistant through the change in the recording state. In this way, Visible can call the initialization interface of the ASR engine through voicekit again to initialize the ASR engine again and provide services for Visible.

[0086] In the above scenario where Visible and Talk is started first, the voice assistant is started later, then exits, and then Visible and Talk is restored, in order to switch the ASR engine between Visible and Talk and the voice assistant, Visible and Talk and the voice assistant need to release the ASR engine and initialize the ASR engine multiple times respectively. In the process of releasing and initializing the ASR engine, the voicekit needs to be repeatedly unregistered and initialized.

[0087] However, voicekit includes multiple modules such as data collection, data processing, intent understanding, intent execution, and capability connectors, and is bound to engines such as ASR, NLU, and DM through capability connectors. In the above-mentioned process of seeing and speaking, voice assistant deregistration, and initializing voicekit, not only the module in voicekit responsible for interfacing with the ASR engine is repeatedly unbound and bound to the ASR engine, but also other modules in voicekit other than the ASR engine are repeatedly unbound and bound to other engines. However, except for the ASR engine, the use process of seeing and speaking and voice assistant for other engines is the same. Therefore, in the scenario of seeing and speaking starting, voice assistant starting and then exiting, and then restoring seeing and speaking, in the steps of repeatedly deregistering and initializing voicekit, the process of repeatedly unbinding and binding other modules in voicekit other than the ASR engine with other engines is wasteful. This may increase the delay during service switching and cause unnecessary power consumption.

[0088] Based on this, an embodiment of the present application provides a method for calling a speech recognition engine, which can be applied to an electronic device. The electronic device has a first voice control function, and a first voice control module corresponding to the first voice control function is connected to the ASR engine through voicekit. In response to the first event, when voicekit is in the startup state, the first voice control module does not need to reinitialize voicekit, but can directly initiate a first call request for the initialization interface to the ASR engine through voicekit. Afterwards, the ASR engine will initialize the first service data corresponding to the first voice control module in response to the first call request. Afterwards, the ASR engine will provide services for the first voice control function.

[0089] In this solution, when the first voice control function needs to call the ASR engine, it is detected that the voicekit is in the startup state. In this way, the voicekit will no longer be deregistered and reinitialized. Instead, the initialization interface of the ASR engine can be directly called through the voicekit to achieve the effect of the first voice control function calling the ASR engine. In this way, in the scenario where two or more voice control functions using the same ASR engine are started one after another, repeated deregistration and initialization of the voicekit can be avoided, thereby avoiding repeated unbinding and binding between other modules in the voicekit other than the ASR engine and other engines. This reduces the delay when the ASR engine switches services and reduces unnecessary power consumption.

[0090] The calling method of the speech recognition engine provided in the embodiment of the present application can be applied to an electronic device 100 including a microphone. The electronic device 100 may include a mobile phone, a tablet computer, a laptop computer, a personal computer (PC), an ultra-mobile personal computer (UMPC), a handheld computer, a netbook, a smart home device (e.g., a smart TV, a smart screen, a large screen, a smart speaker, a smart air conditioner, etc.), a personal digital assistant (PDA), a wearable device (e.g., a smart watch, a smart bracelet, etc.), a vehicle-mounted device, a virtual reality device, etc., and the embodiment of the present application does not impose any restrictions on this.

[0091] like Figure 7 The figure shows a schematic diagram of the structure of an electronic device 100 provided by an embodiment of the present application. The electronic device 100 may include a processor 210, an external memory interface 220, an internal memory 221, a universal serial bus (USB) interface 230, a charging management module 240, a power management module 241, a battery 242, an antenna 1, an antenna 2, a mobile communication module 250, a wireless communication module 260, an audio module 270, a sensor module 280, a button 290, a motor 291, a camera 292, a display screen 293, and a subscriber identification module (SIM) card interface 294, etc. Among them, the sensor module 280 may include a pressure sensor 280A, a touch sensor 280B, etc.

[0092] It is to be understood that the structure illustrated in the embodiment of the present application does not constitute a specific limitation on the electronic device 100. In other embodiments of the present application, the electronic device 100 may include more or fewer components than shown in the figure, or combine some components, or split some components, or arrange the components differently. The components shown in the figure may be implemented in hardware, software, or a combination of software and hardware.

[0093] The processor 210 may include one or more processing units, for example: the processor 210 may include an application processor (application processor, AP), a modem processor, a graphics processor (graphics processing unit, GPU), an image signal processor (image signal processor, ISP), a controller, a memory, a video codec, a digital signal processor (digital signal processor, DSP), a baseband processor, and / or a neural-network processing unit (neural-network processing unit, NPU), etc. Among them, different processing units can be independent devices or integrated in one or more processors. For example, the processor 210 is used to execute the calling method of the speech recognition engine in the embodiment of the present application.

[0094] The controller may be the nerve center and command center of the electronic device 100. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0095] The processor 210 may also be provided with a memory for storing instructions and data. In some embodiments, the memory in the processor 210 is a cache memory. The memory may store instructions or data that the processor 210 has just used or cyclically used. If the processor 210 needs to use the instruction or data again, it may be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 210, and thus improves the efficiency of the system.

[0096] The USB interface 230 is an interface that complies with USB standard specifications, and may specifically be a MiniUSB interface, a MicroUSB interface, a USB Type C interface, etc. The USB interface 230 may be used to connect a charger to charge the electronic device 100, and may also be used to transmit data between the electronic device 100 and a peripheral device.

[0097] The external memory interface 220 can be used to connect an external memory card, such as a MicroSD card, to expand the storage capacity of the electronic device 100. The external memory card communicates with the processor 210 through the external memory interface 220 to implement a data storage function, such as storing music, video and other files in the external memory card.

[0098] The internal memory 221 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 210 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the internal memory 221. The internal memory 221 may include a program storage area and a data storage area. Among them, the program storage area may store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.).

[0099] In addition, the internal memory 221 may include a high-speed random access memory and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0100] The charging management module 240 is used to receive charging input from a charger. The charger may be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 240 may receive charging input from the wired charger through the USB interface 230.

[0101] The power management module 241 is used to connect the battery 242, the charging management module 240 and the processor 210. The power management module 241 receives input from the battery 242 and / or the charging management module 240 to power the processor 210, the internal memory 221, the external memory, the display screen 293, the camera 292, and the wireless communication module 260.

[0102] In some other embodiments, the power management module 241 may also be disposed in the processor 210. In some other embodiments, the power management module 241 and the charging management module 240 may also be disposed in the same device.

[0103] The wireless communication function of the electronic device 100 can be implemented through the antenna 1, the antenna 2, the mobile communication module 250, the wireless communication module 260, the modem processor and the baseband processor.

[0104] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in electronic device 100 can be used to cover a single or multiple communication frequency bands. Different antennas can also be reused to improve the utilization of antennas. For example, antenna 1 can be reused as a diversity antenna for a wireless local area network. In some other embodiments, the antenna can be used in combination with a tuning switch.

[0105] The mobile communication module 250 can provide solutions for wireless communications including 2G / 3G / 4G / 5G, etc., applied to the electronic device 100. The mobile communication module 250 may include at least one filter, a switch, a power amplifier, a low noise amplifier (LNA), etc. The mobile communication module 250 can receive electromagnetic waves from the antenna 1, and filter, amplify, etc. the received electromagnetic waves, and transmit them to the modulation and demodulation processor for demodulation. The mobile communication module 250 can also amplify the signal modulated by the modulation and demodulation processor, and convert it into electromagnetic waves for radiation through the antenna 1.

[0106] The wireless communication module 260 can provide wireless communication solutions including wireless local area networks (WLAN) (such as Wi-Fi), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), infrared (IR), etc., which are applied to the electronic device 100. The wireless communication module 260 can be one or more devices integrating at least one communication processing module. The wireless communication module 260 receives electromagnetic waves via the antenna 2, modulates the frequency of the electromagnetic wave signal and filters it, and sends the processed signal to the processor 210. The wireless communication module 260 can also receive the signal to be sent from the processor 210, modulate the frequency of it, amplify it, and convert it into electromagnetic waves for radiation through the antenna 2.

[0107] In some embodiments, antenna 1 of electronic device 100 is coupled to mobile communication module 250, and antenna 2 is coupled to wireless communication module 260, so that electronic device 100 can communicate with the network and other devices through wireless communication technology.

[0108] The electronic device 100 can implement audio functions such as music playing and recording through the audio module 270 and the application processor.

[0109] The audio module 270 is used to convert digital audio signals into analog audio signals for output, and is also used to convert analog audio inputs into digital audio signals. The audio module 270 can also be used to encode and decode audio signals. In some embodiments, the audio module 270 can be arranged in the processor 210, or some functional modules of the audio module 270 can be arranged in the processor 210.

[0110] The pressure sensor 280A is used to sense the pressure signal and can convert the pressure signal into an electrical signal. In some embodiments, the pressure sensor 280A can be set on the display screen 293. There are many types of pressure sensors 280A, such as resistive pressure sensors, inductive pressure sensors, capacitive pressure sensors, etc. A capacitive pressure sensor can be a parallel plate including at least two conductive materials. When a force acts on the pressure sensor 280A, the capacitance between the electrodes changes. The electronic device 100 determines the intensity of the pressure based on the change in capacitance. When a touch operation acts on the display screen 293, the electronic device 100 detects the intensity of the touch operation based on the pressure sensor 280A. The electronic device 100 can also calculate the position of the touch based on the detection signal of the pressure sensor 280A.

[0111] The touch sensor 280B is also called a "touch panel". The touch sensor 280B can be arranged on the display screen 293, and the touch sensor 280B and the display screen 293 form a touch screen, also called a "touch screen". The touch sensor 280B is used to detect touch operations acting on or near it. The touch sensor can pass the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 293. In other embodiments, the touch sensor 280B can also be arranged on the surface of the electronic device 100, which is different from the position of the display screen 293.

[0112] The key 290 includes a power key, a volume key, etc. The key 290 may be a mechanical key or a touch key. The electronic device 100 may receive key input and generate key signal input related to user settings and function control of the electronic device 100.

[0113] Motor 291 can generate vibration prompts. Motor 291 can be used for incoming call vibration prompts, and can also be used for touch vibration feedback.

[0114] The electronic device 100 implements the display function through a GPU, a display screen 293, and an application processor. The GPU is a microprocessor for image processing, which connects the display screen 293 and the application processor. The GPU is used to perform mathematical and geometric calculations for graphics rendering. The processor 210 may include one or more GPUs that execute program instructions to generate or change display information.

[0115] The display screen 293 is used to display images, videos, etc. In some embodiments, the electronic device 100 may include 1 or N display screens 293 , where N is a positive integer greater than 1.

[0116] The camera 292 is used to capture still images or videos. In some embodiments, the electronic device 100 may include 1 or N cameras 292, where N is a positive integer greater than 1.

[0117] The SIM card interface 294 is used to connect a SIM card. The SIM card can be connected to or disconnected from the electronic device 100 by inserting or removing the SIM card interface 294. The electronic device 100 can support 1 or N SIM card interfaces, where N is a positive integer greater than 1.

[0118] The calling methods of the speech recognition engine in the following embodiments can all be implemented in the electronic device 100 having the above-mentioned hardware structure.

[0119] In the embodiment of the present application, the electronic device 100 is an electronic device that can run an operating system and install application programs. Optionally, the operating system running on the electronic device can be system, system, System, etc.

[0120] In some embodiments, the software system of the electronic device 100 may adopt a layered architecture, an event-driven architecture, a micro-kernel architecture, or a cloud architecture. Taking the system as an example, the software structure of the electronic device 100 is exemplarily described.

[0121] The layered architecture divides the software into several layers, each with a clear role and division of labor. The layers communicate with each other through software interfaces. The system is divided into four layers, from top to bottom: application layer, application framework layer, Android runtime and system library, and kernel layer.

[0122] like Figure 8 As shown, the application layer may include a series of application packages APK. For example, applications such as camera, recording, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video call, short message, etc. The application layer also includes a voice processing engine for parsing, identifying and processing voice. In an embodiment of the present application, the application layer includes a voice control APK, which is used to provide a voice control function (for example, a visible and speakable function and a voice assistant function) of the electronic device 100. Exemplarily, the voice control APK includes a visible and speakable APK and a voice assistant application; the visible and speakable APK is used to provide the visible and speakable function of the electronic device 100, and the voice assistant application is used to provide the voice assistant function of the electronic device 100; wherein, the visible and speakable APK includes a service presentation module, a service processing module, a voice processing module, etc.

[0123] Among them, the business presentation module is used to provide information displayed to the user on the user interface. For example, it provides prompt information displayed on the display interface of the electronic device 100. The business processing module is used to control the relevant process of recording voice when you can see and speak. For example, it controls the opening and closing of the visible and speaking function, controls whether the visible and speaking function is available or unavailable, etc. The business processing module may include a recording management module, a status management module, a monitoring module, etc. The monitoring module is used to monitor whether other applications occupy the microphone. The recording management module is used to control the opening and closing of the recording channel corresponding to the visible and speaking function; for example, according to the visible and speaking switch being turned on or off, the recording channel corresponding to the visible and speaking function is turned on or off. The status management module is used to manage the status of the visible and speaking function, such as turned on, turned off, temporarily unavailable, etc.

[0124] The voice processing module is used to recognize and process the collected user voice. For example, the voice processing module calls the voice processing engine to perform text parsing, semantic understanding and recognition on the audio stream collected by the microphone, and converts the user voice into voice commands. The voice processing module sends the voice commands to the multi-mode control module of the application framework layer, and the multi-mode control module distributes the voice commands to the corresponding application for execution.

[0125] The application framework layer provides an application programming interface (API) and a programming framework for the applications in the application layer. The application framework layer includes some predefined functions.

[0126] like Figure 8 As shown, the application framework layer may include a window manager, a content provider, a view system, a resource manager, an activity manager service (AMS), an application manager (PMS), a multi-mode control module, an audio module, and the like.

[0127] The window manager is used to manage window programs. The window manager can obtain the display screen size, determine whether there is a status bar, lock the screen, capture the screen, etc.

[0128] Content providers are used to store and retrieve data and make it accessible to applications. The data can include videos, images, audio, calls made and received, browsing history and bookmarks, phone books, etc.

[0129] The view system includes visual controls, such as controls for displaying text, controls for displaying images, etc. The view system can be used to build applications. A display interface can be composed of one or more views. For example, a display interface including a text notification icon can include a view for displaying text and a view for displaying images.

[0130] The resource manager provides various resources for applications, such as localized strings, icons, images, layout files, video files, and so on.

[0131] AMS is mainly responsible for the startup, switching, and scheduling of the four major components in the system, as well as the management and scheduling of application processes. Its responsibilities are similar to the process management and scheduling modules in the operating system. When a process or component is started, the request will be passed to AMS through the Binder communication mechanism, and AMS will then handle it uniformly.

[0132] PMS handles package management related tasks, such as application installation, application uninstallation, etc. It is also used to provide all the information of an application, such as startup, shutdown, etc.

[0133] The multi-mode control module is used to manage the execution of voice commands, for example, sending voice commands to applications so that the applications execute the voice commands.

[0134] The audio module is used to handle audio-related functions, such as recording audio, playing audio, etc. It includes a microphone, which is used to manage the related processes of audio recording.

[0135] The system library may include multiple functional modules, such as surface manager, media library, 3D graphics processing library (such as OpenGL ES), 2D graphics engine (such as SGL), etc.

[0136] The surface manager is used to manage the display subsystem and provide the fusion of 2D and 3D layers for multiple applications.

[0137] The media library supports playback and recording of a variety of commonly used audio and video formats, as well as static image files, etc. The media library can support a variety of audio and video encoding formats, such as: MPEG4, H.264, MP3, AAC, AMR, JPG, PNG, etc.

[0138] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0139] A 2D graphics engine is a drawing engine for 2D drawings.

[0140] Android runtime is responsible for the scheduling and management of the Android system. Android runtime includes core libraries and virtual machines.

[0141] The core library consists of two parts: one is the function that needs to be called by the Java language, and the other is the Android core library.

[0142] The application layer and the application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and the application framework layer as binary files. The virtual machine is used to perform functions such as object life cycle management, stack management, thread management, security and exception management, and garbage collection.

[0143] The kernel layer is the layer between hardware and software. The kernel layer can include display drivers, sensor drivers, audio drivers, Wi-Fi drivers, etc. The kernel layer also includes a wake-up word detection module, which is used to detect whether the wake-up word is received based on the audio collected by the microphone. The kernel layer also includes an audio digital signal processor (ADSP), which is used to distribute and process the audio collected by the microphone, such as establishing a recording channel for the application and distributing audio streams to the application.

[0144] Combination Figure 8 , Fig. 9 A schematic diagram of the interaction of various modules of an electronic device when controlling an electronic device by voice provided in an embodiment of the present application is shown.

[0145] See also Fig. 9 , Visible Speak and Voice Assistant both need to connect to the voice control engine through voicekit. Specifically, Visible Speak and Voice Assistant can be connected to the data collection, data understanding, intent understanding and intent processing modules in voicekit respectively, and can be bound to the ASR engine, NLU engine and DM engine in the voice control engine through the capability connector in voicekit. Therefore, Visible Speak and Voice Assistant can use all engines in the voice control engine to realize related voice recognition and control functions.

[0146] The following combination Fig. 9, taking Visible is Talk as an example, the process of voice control of electronic devices based on Visible is Talk is explained: when using Visible is Talk to realize the control of electronic devices, the screen content module of Visible is Talk obtains screen elements from the screen elements of the framework layer for text analysis and layout analysis, and then transmits the analyzed screen context to the instruction distribution module. After Visible is Talk collects audio data, it can transmit the audio data to the voice control engine through VoiceKit for data understanding, intent understanding and intent processing, so as to determine the voice instructions to be executed in the audio data. Then, the voice control engine transmits the determined voice instructions to the instruction distribution module of Visible is Talk, and the instruction distribution module determines the type of voice instructions (control click class or general instruction class). Then, the instruction distribution module can transmit the determined type of voice instructions to the instruction arrangement and execution module for preprocessing, adaptation or conversion. Finally, the instruction arrangement and execution module transmits the voice instructions to the multi-module interaction framework, and executes the operation corresponding to the voice instructions through the multi-modal interaction framework.

[0147] The following will describe in detail the calling method of the voice recognition engine provided in the embodiment of the present application in conjunction with the accompanying drawings. The following is an example in which the electronic device is a mobile phone, and the mobile phone first starts the second voice control function and then starts the first voice control function. In addition, in this embodiment, the second voice control function is see-and-speak, and the first voice control function is a voice assistant. In this embodiment, the mobile phone may include: a control module corresponding to see-and-speak, a control module corresponding to the voice assistant, a voice kit voicekit and an ASR engine. Please refer to Fig.10 , the method comprising:

[0148] S401. It can be seen that a voice control trigger event 1 is received.

[0149] The voice control trigger event 1 is used to trigger the start of the second voice control function (in this embodiment, visible and audible).

[0150] In some embodiments, the voice control trigger event 1 may specifically be a see-and-speak start event. The see-and-speak start event may specifically correspond to a user manually turning on a see-and-speak control function. Figure 3 The switch 107 in the setting interface 105 is shown to turn on the visible control function.

[0151] In other embodiments, the above-mentioned voice control trigger event 1 may also be visible, that is, monitoring the exit of other voice functions. The voice function may be a voice control function such as a voice assistant, an audio or video call, or a recording function, etc. Figure 6It can be seen from the process that after Visible is Talkable is started, if the user triggers the voice assistant, the voice control function based on Visible is Talkable will be temporarily interrupted. After the voice assistant exits, Visible is Talkable can resume voice control again. Therefore, in this embodiment, the above-mentioned voice control trigger event 1 may also be a voice assistant exit event. It is understandable that in other embodiments, Visible is Talkable may also be resumed after voice control is interrupted by other voice functions; in this embodiment, voice control trigger event 1 may specifically be an exit event of the corresponding voice function.

[0152] exist Fig.10 In the illustrated embodiment, before S401, the mobile phone has not activated the see-and-speak function, and therefore, the voice control triggering event 1 in S401 is a see-and-speak activation event.

[0153] S402. In response to voice control triggering event 1, start the see-and-say voice control function.

[0154] In some embodiments, during the process of starting See and Speak, the mobile phone will establish a recording channel corresponding to See and Speak. The audio stream collected by the microphone will be distributed to the See and Speak application for processing through the recording channel corresponding to the See and Speak function. In addition, after establishing the recording channel corresponding to See and Speak, See and Speak will call the voice control engine to implement voice control based on See and Speak.

[0155] S403. It can be seen that the call request 1 is sent to the initialization interface of the ASR engine through voicekit.

[0156] As can be seen from the above description, Visible and Talkable needs to be connected to the ASR engine through VoiceKit. Therefore, when Visible and Talkable requests to call the ASR engine, it needs to send a call request for the initialization interface of the ASR engine to the ASR engine through VoiceKit.

[0157] It should be noted that when the mobile phone implements voice control based on See and Speak, See and Speak needs to use the ASR engine, NLU engine, DM engine and other engines in the voice control function. Therefore, in some embodiments, after See and Speak is started, in addition to sending a call request 1 to the ASR engine through VoiceKit, it is also necessary to send a call request to other engines in the voice control engine except the ASR engine through VoiceKit.

[0158] In the embodiment of the present application, the ASR engine is not started before Visible is Talked is started. That is, before Visible is Talked requests to call the ASR engine, VoiceKit is not started. Therefore, after Visible is Talked sends a call request 1 to VoiceKit, VoiceKit itself needs to be initialized first, such as S404.

[0159] S404. In response to calling request 1, voicekit is initialized.

[0160] Among them, the specific implementation process of voicekit initialization can refer to the introduction in the relevant technology and will not be repeated in the embodiments of this application.

[0161] S405. voicekit sends a call request 1 for the initialization interface of the ASR engine to the ASR engine.

[0162] After the initialization is completed, the voicekit can initiate a call request to the initialization interface of the ASR engine. In some embodiments, the voicekit can forward the call request 1 sent by the ASR engine.

[0163] In some embodiments, the initialization interface of the ASR engine may also be referred to as an init interface.

[0164] S406. The ASR engine binds to the voicekit in response to the call request 1.

[0165] Combination Fig. 9 It can be known that the ASR engine is bound to the voicekit, which may specifically include: the module in the voicekit responsible for connecting with the ASR engine is bound to the ASR engine through a capability connector.

[0166] S407. The ASR engine initializes the service data to be visible, that is, the corresponding service data 1.

[0167] Since the call request 1 is initiated for the initialization interface of the ASR engine, the ASR engine will be initialized in response to the call request 1. In some embodiments, S407 may specifically include: registering the context, hot words, etc. corresponding to the visible-to-speak control function in the ASR engine. So that the ASR engine can perform voice recognition on the audio data sent after the visible-to-speak function in combination with the context and hot words, etc., and determine the voice instructions therein. It can be understood that after the ASR engine initializes the service data 1 corresponding to the visible-to-speak function, the ASR engine can provide services for the visible-to-speak function.

[0168] After the ASR engine is initialized in response to the call request of the voice control function, the ASR engine may write the relevant information of the voice control function currently calling the ASR engine in the cache of the mobile phone. In some embodiments, the relevant information of the voice control function written by the ASR engine in the cache may specifically include: function identification information of the voice control function (such as id, corresponding package name, etc.). In this embodiment, the above S407 may specifically include: the ASR engine writes the relevant information that can be seen and said in the cache.

[0169] Further, in some embodiments, after the ASR engine writes the relevant information of the voice control function currently calling the ASR engine in the cache, it can also generate corresponding call parameters based on the voice control function. It can be understood that the call parameters can be used to characterize the voice control function of the ASR engine currently being used, that is, the current user of the ASR engine. In some embodiments, after the ASR engine writes the relevant information of Visible and Speakable in the cache, it will also generate call parameters corresponding to Visible and Speakable, and save the call parameters as the current call parameters of the ASR engine. In some embodiments, the above-mentioned call parameters can specifically be handlers.

[0170] S408. The voice assistant receives voice control trigger event 2.

[0171] In the embodiment of the present application, the first voice control function is a voice assistant, and therefore, the above-mentioned voice control trigger event 2 is the first voice control trigger event.

[0172] In some embodiments, the above-mentioned voice control trigger event 2 may specifically be a voice assistant startup event. The voice assistant startup event may correspond to the user turning on the voice assistant control function. Exemplarily, the voice assistant startup event may specifically correspond to any of the following events: the mobile phone detects a preset operation on a preset button, or the mobile phone detects a wake-up word for the voice assistant, or the mobile phone detects a trigger operation on the voice assistant function icon.

[0173] Similar to the first event, after the voice assistant is started, it may also be interrupted by other voice functions (such as voice control functions such as see-and-speak, audio and video calls, or recording functions, etc.). Afterwards, after other voice functions exit, the voice assistant can resume voice control. Therefore, in some embodiments, the above-mentioned voice control trigger event 2 may also correspond to the exit event of other voice functions.

[0174] exist Fig.10 In the illustrated embodiment, before S408 , the mobile phone does not start the voice assistant, and therefore, the voice control trigger event 2 in S408 is a voice assistant start event.

[0175] S409. In response to voice control triggering event 2, start the voice control function of the voice assistant.

[0176] In some embodiments, when the voice assistant is started, the mobile phone will establish a recording channel corresponding to the voice assistant function. The audio stream collected by the microphone will be distributed to the voice assistant application for processing through the recording channel corresponding to the voice assistant function.

[0177] S410. The voice assistant sends a call request 2 for the initialization interface of the ASR engine to the ASR engine through the voicekit.

[0178] In the embodiment of the present application, the first voice control function is a voice assistant, and therefore, the above-mentioned call request 2 is the first call request.

[0179] It should be noted that when a mobile phone implements voice control based on a voice assistant, the voice assistant needs to use the ASR engine, NLU engine, DM engine and other engines in the voice control function. Therefore, in some embodiments, after the voice assistant is started, in addition to sending a call request 2 to the ASR engine through voicekit, it is also necessary to send a call request to other engines in the voice control engine except the ASR engine through voicekit.

[0180] S411. The ASR engine responds to the call request 2, cancels the service data 1 corresponding to the visible and audible function, and initializes the service data to the service data 2 corresponding to the voice assistant.

[0181] In the embodiment of the present application, the first voice control function is a voice assistant, so the above service data 2 is the first service data. In some embodiments, the ASR engine cancels the service data 1 corresponding to the visible and speakable function, that is, the visible and speakable function releases the ASR engine.

[0182] It can be understood that after the ASR engine initializes the service data of the ASR engine to the service data 2 corresponding to the voice assistant, the ASR engine can provide services for the voice assistant.

[0183] Furthermore, since the ASR engine in S411 deregisters the service data corresponding to Visible and Speakable in response to the call request of other functions for the initialization interface, rather than deregistering in response to the call request of Visible and Speakable itself for the deregistration interface of the ASR engine, in some embodiments, after the ASR engine switches from Visible and Speakable to the voice assistant in response to the call request 2, the ASR engine can suspend Visible and Speakable.

[0184] In some embodiments, when the ASR engine cancels the service data 1 corresponding to Visible and Speak, it will not delete the relevant information of Visible and Speak stored in the cache of the ASR engine. So that after the voice assistant exits, the ASR engine responds to the call request re-initiated by Visible and Speak based on the relevant information of Visible and Speak stored in the cache and restores the call of Visible and Speak to the ASR engine. The specific process of the ASR engine responding to the call request re-initiated by Visible and Speak after the voice assistant exits and restoring the call of Visible and Speak to the ASR engine will be described in detail in the following embodiments.

[0185] At the same time, in some embodiments, after the ASR engine initializes the service data to service data 2 corresponding to the voice assistant in S411, the above method may also include: the ASR engine inserts relevant information of the voice assistant into the cache.

[0186] In addition, although the ASR engine will not delete the relevant information of Visible and Speakable in the cache, since the voice control function that currently calls the ASR engine is switched to the voice assistant, the ASR engine needs to generate new call parameters based on the voice assistant. In some embodiments, after the ASR engine inserts the relevant information of the voice assistant in the cache, it will also generate new call parameters based on the voice assistant. In some embodiments, the ASR engine saves the latest call parameters in the cache, that is, the ASR engine updates the current call parameters from the second call parameters corresponding to Visible and Speakable to the first call parameters corresponding to the voice assistant.

[0187] In some embodiments, the ASR engine can specifically cache the relevant information of the current user of the ASR engine in the state queue. Exemplarily, the ASR engine can insert the voice assistant or the relevant information that can be seen and said at the head of the state queue.

[0188] It can be understood that in the above S411, the ASR engine initializes the service data of the ASR engine to the service data corresponding to the voice assistant, which can be specifically executed after the ASR engine completes the cancellation of the visible and corresponding service data.

[0189] In S410, for the ASR engine, when receiving the call request 2 sent by the voice assistant through the voicekit, the ASR engine is providing services for Visible is Speakable. That is, the current ASR engine is in an occupied state. In some embodiments, if the ASR engine receives a call request from other functions during use, it can first determine whether the switching condition is currently met. If so, the ASR engine is allowed to switch from the current function to other functions; otherwise, the switch is not allowed.

[0190] In some embodiments, after the ASR engine in S410 receives the call request 2, before S411, the method further includes: determining whether a preset switching condition is currently met. In this embodiment, S411 may specifically include: if the switching condition is met, in response to the call request 2, the ASR engine cancels the second access data corresponding to the visible and audible function, and initializes the service data to the service data 2 corresponding to the voice assistant.

[0191] Further, in some embodiments, the ASR engine determines whether the switching condition is currently met, which may specifically include: determining whether the voice control function (such as Visible and Speakable) currently using the ASR engine allows switching. If Visible and Speakable allows switching, it is determined that the switching condition is met; at this time, the ASR engine can be deregistered and reloaded. If Visible and Speakable does not allow switching, it is determined that the switching condition is not met; at this time, the ASR engine can directly ignore the call request initiated later, or the ASR engine can return a response message of initialization failure to the function (such as voice assistant) that initiated the call request later.

[0192] Alternatively, the mobile phone can also pre-set a switchable whitelist that allows switching in the ASR engine. Functions that are allowed to switch can be set in the switchable whitelist. In this embodiment, the above-mentioned ASR engine determines whether the switching conditions are currently met, which can specifically include: determining whether the function currently using the ASR engine is in the switchable whitelist. Further, in this embodiment, the above-mentioned S411 can specifically include: in the case where Visible is Speakable is included in the switchable whitelist, in response to call request 2, the ASR engine cancels the service data 1 corresponding to Visible is Speakable, and initializes the service data to the service data 2 corresponding to the voice assistant.

[0193] Alternatively, for multiple functions that support calling the ASR engine, the mobile phone can pre-set priorities and store the pre-set priority information in the ASR engine. Among them, functions with higher priorities can call the ASR engine first. Therefore, in some embodiments, the ASR engine determines whether the switching conditions are currently met, and may also include: the ASR engine can determine, based on the pre-stored priority information, whether the priority of the function that initiates the call request later (such as a voice assistant) is higher than the function that initiates the call request earlier (such as Visible and Speakable). Further, in this embodiment, the above S411 may specifically include: if the priority of the voice assistant is higher than the priority of Visible and Speakable, then in response to the call request 2, the ASR engine cancels the service data 1 corresponding to Visible and Speakable, and initializes the service data to the service data 2 corresponding to the voice assistant.

[0194] In other embodiments, if the priority of the function that initiates the call request later is lower than the priority of the function that initiates the call request earlier, the ASR engine may ignore the call request initiated later; or, the ASR engine returns a response message of initialization failure to the function that initiates the call request later. For example, if the ASR engine determines that the priority of the voice assistant is lower than the priority of See and Speak, the ASR engine may ignore the call request 2 sent by the voice assistant through VoiceKit; or, the ASR engine returns a response message of initialization failure to the voice assistant.

[0195] In the technical solution provided by the embodiment of the present application, for the scenario where the ASR engine receives a new call request while in use, the ASR engine needs to allow the function currently using the ASR engine to switch to the function that initiates the call request later only when the switching conditions are met. In this way, it is possible to avoid the problem of false triggering and false switching, which causes the voice control function to be unable to use the ASR engine normally.

[0196] S412. In response to the first exit event, the voice assistant exits, and the voice assistant sends a call request 3 for the logout interface of the ASR engine to the ASR engine through voicekit.

[0197] In some embodiments, the deregistration interface of the ASR engine may also be referred to as the destroy interface of the ASR engine.

[0198] In some cases, the user may have triggered the voice assistant by mistake. In this case, the user can manually exit the voice assistant after discovering that the voice assistant is turned on. Exemplarily, the user can exit the voice assistant by clicking the back control / back button / exit control / exit button, or by performing a preset back gesture on the display. Therefore, in some embodiments, the first exit event may specifically correspond to the mobile phone detecting the user's exit operation on the voice assistant.

[0199] Alternatively, after the voice assistant is started, the user can also control the mobile phone to exit the voice assistant by voice control. For example, after the voice assistant is started, the user can say a voice command such as "exit" to control the mobile phone to exit the voice assistant; the voice assistant will exit the voice assistant in response to the voice command. Therefore, in other embodiments, the first exit event may also correspond to the detection of a voice command corresponding to the exit operation.

[0200] Alternatively, in some mobile phones, the voice assistant can also set an automatic exit mechanism. For example, if the voice assistant does not receive valid audio data within a period of time after it is started, the voice assistant can automatically exit. In other embodiments, the first exit event can also correspond to the voice assistant exiting when no valid audio data is received within a preset time period.

[0201] It can be understood that the above embodiments are only partial examples of the first exit event. In other embodiments, the first exit event may also be other events.

[0202] S413. The ASR engine responds to the call request 3 and cancels the service data 2 corresponding to the voice assistant.

[0203] As can be seen from S412, the call request 3 is sent by the voice assistant itself to the ASR engine through voicekit. Therefore, in some embodiments, in response to the call request 3, the ASR engine can delete the relevant information of the voice assistant in the cache while canceling the service data 2 corresponding to the voice assistant.

[0204] In addition, in response to the first exit event, the voice assistant exits, which means that the voice assistant will stop using the recording permission. Therefore, in some embodiments, a recording state change event is also triggered, such as S414.

[0205] S414. The voice assistant triggers a recording status change event.

[0206] It is understandable that the mobile phone can execute S412-S413 first and then execute S414, or can execute S414 first and then execute S412-S413, or can execute S412-S413 and S414 simultaneously. In the embodiment of the present application, the order of executing S412-S413 and S414 is not limited.

[0207] It can be seen from the description of the above embodiment that the recording status change event of S414 is triggered by the exit of the voice assistant. The recording status change event can indicate that the voice assistant stops using the recording permission. At the same time, the exit of the voice assistant also means that the application assistant will stop using the ASR engine. Combined with the description of S401-S407, it can be seen that before the voice assistant is started, Visible is started; and after the voice assistant is started, Visible is temporarily interrupted. That is to say, in the process of performing voice control based on the voice assistant, Visible is in a state of being suspended by the ASR engine. Therefore, when the voice assistant exits, Visible can resume using the ASR engine to restore the voice control function based on Visible.

[0208] S415. It can be seen that in response to the recording state change event, a call request 4 for the initialization interface of the ASR engine is sent to the ASR engine through the voicekit.

[0209] It can be seen from the description of the above embodiment that the voice assistant triggers a voice state change event when exiting. Therefore, it can be seen that after detecting the recording state change event, it can be determined that the voice assistant has exited. After that, it can be seen that the ASR engine can be re-called. In this embodiment, the recording state change event belongs to the above-mentioned voice control trigger event 1. Therefore, in response to the recording state change event, it can be seen that the above-mentioned call request 4 can be sent to the ASR engine through voicekit.

[0210] It can be seen that when sending a call request to the ASR engine through voicekit again, if voicekit is in the started state, then it can be seen that the call request can be sent directly to the ASR engine through voicekit. If voicekit is not started, it can be seen that sending a call request to the ASR engine through voicekit requires initializing voicekit first.

[0211] In some embodiments, when a recording state change event is detected, Visible is Talkable can first determine whether it has previously called the ASR engine to determine the state of VoiceKit. If Visible is Talkable has previously called the ASR engine, VoiceKit should be in the startup state. At this time, Visible is Talkable can directly try to re-call the initialization interface of the ASR engine through VoiceKit to restore the voice control function based on Visible is Talkable.

[0212] In other embodiments, when the recording state change event is detected, VoiceKit may have been activated by other voice functions. Therefore, in response to the recording state change event, it can be seen that the state of VoiceKit can also be directly queried.

[0213] In the technical solution provided in the embodiment of the present application, it can be seen that when voice control is restored, it can be determined whether voicekit needs to be initialized according to the status of voicekit. If voicekit has been started, it can be seen that there is no need to reinitialize voicekit, but a call request can be directly sent to the ASR engine through voicekit to reinitialize the ASR engine. Therefore, when the ASR engine switches between different voice control functions, the number of times voicekit is deregistered and initialized can be reduced, the delay during service switching can be reduced, and the waste of functions can be reduced.

[0214] S416. The ASR engine determines whether the current engine state is an idle state.

[0215] In some embodiments, the engine state of the ASR engine can be divided into the following states according to whether the ASR engine is being used: idle state, initializing state, occupied state and deregistering state. Among them, the idle state is the state where deregistration is completed.

[0216] In some embodiments, the ASR engine can obtain the current engine state from the state queue. Further, the ASR engine can determine whether the current engine state is an idle state based on whether the relevant information of the voice control function is saved in the state queue. Exemplarily, if the relevant information of the voice control function is saved in the state queue, it means that the current ASR engine is being used by the voice control function; that is, the current engine state of the ASR engine is not an idle state. If the state queue does not save the relevant information of the voice control function, it can be determined that the current ASR engine is not in use and the current engine state is an idle state.

[0217] In an embodiment in which the call parameters corresponding to the current user are generated after the ASR engine is initialized, the ASR engine can also determine whether the current engine state is an idle state by querying whether the ASR engine has saved the call parameters. Exemplarily, if the ASR engine has saved the call parameters, it means that the current ASR engine is being used by the voice control function; that is, the current engine state of the ASR engine is not an idle state. If the ASR engine has not saved the call parameters, it can be determined that the current ASR engine is not in use and the current engine state is an idle state.

[0218] Further, in some embodiments, when the engine state of the ASR engine is in an idle state, calling the initialization interface of the ASR engine is allowed. Moreover, in the idle state, in response to a call request for the initialization interface, the ASR engine can be directly initialized.

[0219] When the engine state of the ASR engine is in the initialization state or the deregistration state, the initialization interface of the ASR engine is not allowed to be called. In some embodiments, in the initialization state or the deregistration state, if a call request for the initialization of the ASR engine is received, the ASR engine can ignore the call request.

[0220] In other embodiments, when the engine state of the ASR engine is occupied, the initialization interface of the ASR engine is allowed to be called. Exemplarily, in the occupied state, the ASR engine responds to the call request for the initialization interface, and will be deregistered and reloaded if certain conditions are met. Taking the scenario of S401-S416 as an example, after the ASR engine initializes the service data 1 corresponding to the visible and speakable, it receives a call request 2 sent by the voice assistant. At this time, in response to the call request 2, the ASR engine can first deregister the service data 1 corresponding to the visible and speakable, and then initialize the service data to the service data 2 corresponding to the voice assistant.

[0221] It can be seen from S412 and S413 that in response to the call request 3 of the voice assistant for the deregistration interface of the ASR engine, the service data 2 corresponding to the voice assistant will be deregistered. Therefore, in some embodiments, when receiving the call request 4 for the initialization interface sent by Visible and Speak through voicekit, it is possible that the voice assistant has not yet completed the deregistration, that is, the ASR engine is in the deregistration state. At this time, the initialization interface of the ASR engine is not allowed to be called. In other words, at this time, the ASR engine cannot initialize in response to the call request 4 of Visible and Speak, and needs to wait until the voice assistant completes the deregistration before responding.

[0222] In some embodiments, if the determination result of S416 is no, the ASR engine may return to continue to obtain the current engine state and determine whether the current engine state is in an idle state.

[0223] Alternatively, in some other embodiments, when receiving the call request 4 sent by the voicekit, the voice assistant has completed releasing the ASR engine, and the ASR engine is in an idle state. At this time, the ASR engine allows the initialization interface of the ASR engine to be called. That is, the ASR engine can directly respond to the call request 4 and reinitialize, such as S417.

[0224] In some embodiments, after the ASR engine responds to the call request 3 and cancels the service data 2 corresponding to the voice assistant, since the state queue still stores the relevant information of Visible is Speak. Therefore, the ASR engine will not immediately unbind from the module responsible for the ASR engine in VoiceKit, but can wait for Visible is Speak to resend the call request for the initialization interface. In this way, the problem that the ASR engine is unbound from VoiceKit and Visible is Speak cannot resume calling the ASR engine can be avoided.

[0225] S417. In response to the call request 4, the ASR engine initializes the service data to be visible, that is, the corresponding service data 1.

[0226] In some embodiments, after the above S417, the ASR engine will regenerate and save the call parameters corresponding to the visible-to-speak.

[0227] In the technical solution provided in the embodiment of the present application, for the scenario where Visible and Speak is started first, the voice assistant is started later and then exited, and then Visible and Speak is restored, when the ASR engine has been started, the voice assistant and Visible and Speak can both send call requests directly to the ASR engine through voicekit. The ASR engine will also respond to the call request in combination with the current engine status. Because in this process, voicekit does not need to be repeatedly deregistered and initialized. In this way, in the process of the ASR engine switching services between Visible and Speak and the voice assistant, the repeated unbinding and binding of the modules responsible for the non-ASR engine in voicekit can be avoided. The ASR recognition engine is deregistered and rebuilt separately, and the shared parts in voicekit are retained. Thereby reducing the delay of the ASR engine in the service switching process, improving the service switching efficiency, and reducing unnecessary functional waste.

[0228] In some cases, it is possible that after a voice control function is started, a startup event for the voice control function is detected. Taking the voice assistant as an example, after detecting the wake-up word of the voice assistant, the mobile phone can start the voice assistant, and the ASR engine will be called after the voice assistant is started. After that, if the mobile phone detects a preset operation for a preset button, it may trigger the voice assistant to start again. For the voice assistant, after the ASR engine has been started and called, detecting a preset operation on a preset button may trigger the step of re-calling the ASR engine. However, when the preset operation on the preset button is detected, the ASR engine may still be in use by the voice assistant.

[0229] At this time, if in response to the preset operation of the preset button, the voice assistant executes the process of calling the ASR engine again, such as sending a call request 5 for the initialization interface to the ASR engine through voicekit. Then the ASR engine may respond to the call request 5 and execute the process of deregistering and reinitializing. However, in response to the call request 5, the ASR engine first deregisters the service data 2 corresponding to the voice assistant, and then reinitializes the service data 2 corresponding to the voice assistant. It can be understood that the process executed by the ASR engine in response to the call request 5 is a completely unnecessary step and will waste the power consumption of the mobile phone.

[0230] In other embodiments, take the example of a voice control function Visible and Talkable. After Visible and Talkable starts and calls the ASR engine, it is interrupted by an audio or video call or a recording function. Afterwards, the audio or video call or the recording function exits, which will trigger a recording status change event. Visible and Talkable detects the recording status change event and can resume voice control based on the ASR engine. Then, when Visible and Talkable resumes using the ASR engine for voice control, the ASR engine is still in a state of being used by Visible and Talkable. At this time, if, in response to the recording status change event, Visible and Talkable re-executes the process of calling the ASR engine, it will also cause the ASR engine to perform unnecessary steps of deregistering and reloading.

[0231] As can be seen from the description of the above embodiments, in some embodiments, after the initialization is completed, the ASR engine will also generate a call parameter corresponding to the voice control function currently calling the ASR engine. Therefore, in some embodiments, in order to avoid the same voice control function from repeatedly calling the ASR engine, resulting in a waste of functions, the ASR engine can also add a query interface. The query interface is used to support querying the current call parameters of the ASR engine. In the above example, after the voice assistant detects the preset operation for the preset button after startup, it can query the current call parameters of the ASR engine when it is determined that the ASR engine is in an occupied state. Then, it is determined by the current call parameters whether the voice control function currently using the ASR engine is consistent with itself. If consistent, it means that the current voice assistant is using the ASR engine, so the voice assistant does not need to re-execute the process of calling the ASR engine. If inconsistent, it means that the voice control function currently using the ASR engine is not a voice assistant. At this time, the voice assistant responds to the preset operation for the preset button and needs to execute the process of re-calling the ASR engine.

[0232] The above embodiments are described using a voice assistant as an example. The above scenarios are also applicable to "see and speak". Fig.11 , which shows the process of calling the speech recognition engine in some other embodiments. In this embodiment, after S407 and before S408, the method further includes S501-S503. Fig.11 A detailed description of the method steps shown can be found in the present application Fig.10 The detailed description of the corresponding method steps in the present application will not be repeated here.

[0233] S501. The ASR engine generates a second call parameter corresponding to "visible means audible", and saves the second call parameter as the current call parameter.

[0234] S502. The ASR engine sends a second calling parameter to the visible device.

[0235] S503. It can be seen that the second calling parameter is saved.

[0236] After saving the second call parameter, when the voice control trigger event 1 (such as a recording status change event) is detected again, the current call parameter of the ASR engine can be queried first. By comparing whether the current call parameter of the ASR engine is consistent with the second call parameter, it can be determined whether the voice control function currently using the ASR engine is visible and speakable. Therefore, it can be determined whether the ASR engine needs to be reinitialized.

[0237] It should be noted that after Visible is Talking exits, Visible is Talking will call the ASR engine's logout interface through VoiceKit to release the ASR engine. In some embodiments, Visible is Talking will also delete the second call parameter after releasing the ASR engine. Similarly, for the ASR engine, if no voice control function is currently using the ASR engine, the current call parameter saved by the ASR engine can be deleted.

[0238] Please continue to refer to Fig.11 In this embodiment, similarly, after S411, the ASR engine will regenerate and save new call parameters based on the voice assistant, and return the newly generated call parameters to the voice assistant, such as S504-S506.

[0239] S504. The ASR engine generates a first calling parameter corresponding to the voice assistant, and changes the current calling parameter to the first calling parameter.

[0240] S505. The ASR engine returns the first calling parameter to the voice assistant.

[0241] S506. The voice assistant saves the first calling parameter.

[0242] In addition, in the above embodiments, after successfully calling the ASR engine, the voice control function will receive the call parameters sent by the ASR engine and save the call parameters. Therefore, in some embodiments, the voice control function can determine whether the ASR engine has been called before by querying whether the call parameters are saved. Exemplarily, it can be determined whether the ASR engine has been called before by querying whether the call parameters of the ASR engine are saved.

[0243] In some embodiments, the above S409 may specifically include: in response to the second event, starting the voice assistant, and querying whether the voice assistant has saved the call parameters of the ASR engine. Further, the above S410 may specifically include: in the case where the voice assistant does not save the call parameters from the ASR engine, sending a call request 2 for the initialization interface of the ASR engine to the ASR engine through voicekit.

[0244] Similarly, for Visible is Speaking, after Visible is Speaking detects the recording state change event in S415, it can also first query whether Visible is Speaking saves the calling parameters of the ASR engine to determine whether the ASR engine has called the ASR engine. Fig.10 In the embodiment, before S415, it can be seen that the second calling parameter is indeed saved, so it can be seen that the current calling parameter of the ASR engine needs to be queried to determine whether the ASR engine needs to be reinitialized. In this embodiment, before S415, the above method can also include S507-S509:

[0245] S507. When the second calling parameter is saved in Visible and Talkable, in response to the recording state change event, Visible and Talkable sends a first query request to the ASR engine through VoiceKit.

[0246] S508. In response to the first query request, the ASR engine sends the current call parameters to the visible device.

[0247] S509. It can be seen that the current calling parameter is compared with the second calling parameter to see whether they are consistent.

[0248] In some embodiments, if Visible to Speak compares the current call parameters to be consistent with the second call parameters, then it means that the current ASR engine is consistent with the ASR engine required to be used by Visible to Speak. At this time, Visible to Speak does not need to reinitialize the ASR engine and can use the ASR engine to start speech recognition.

[0249] In other embodiments, if Visible to Speak compares the current call parameters and the second call parameters and finds them inconsistent, it means that the current ASR engine is different from the ASR engine that Visible to Speak needs to use. At this time, Visible to Speak needs to reinitialize the ASR engine before the ASR engine can be used to implement the voice control function based on Visible to Speak. In the embodiment of the present application, since the ASR engine has changed the current call parameters to the first call parameters in S504, the judgment result of S509 is no. In other words, the second call parameters saved by Visible to Speak are inconsistent with the current call parameters.

[0250] Further, when Visible is Speakable determines that the second calling parameter is inconsistent with the current calling parameter, the initialization interface of the ASR engine can be called through VoiceKit to enable the ASR engine to initialize the service data to Visible is Speakable corresponding service data 1.

[0251] In the technical solution provided in the embodiment of the present application, after the ASR engine completes the initialization, it will return the generated call parameters to the voice function (such as Visible to Speak) that is currently using the ASR engine for storage. At the same time, the ASR engine also provides a query interface for call parameters to the outside world, which is used to query the current call parameters of the ASR engine for the voice function that wants to call the ASR engine. For Visible to Speak that wants to call the ASR engine, it can compare the current call parameters with the call parameters that Visible to Speak has previously saved. If the comparison results are consistent, it means that the current ASR engine is consistent with the ASR engine that Visible to Speak needs to use. Then Visible to Speak can directly use the current ASR engine without reinitializing the ASR engine. In this way, the redundant steps of canceling service data 1 and reinitializing the service data to service data 1 can be avoided, thereby reducing power consumption waste.

[0252] In addition, if the comparison result of Visible to Speak on the current call parameters and the previously saved call parameters is inconsistent, it means that the current ASR engine is inconsistent with the ASR engine that Visible to Speak needs to use. In this case, the current ASR engine needs to be deregistered and Visible to Speak reinitializes the ASR engine to implement the voice control function based on Visible to Speak through the ASR engine. In this way, it is ensured that Visible to Speak can use the function of the ASR engine normally to implement voice control.

[0253] In other embodiments, the first voice control function may be Visible and Speak, and the second voice control function may be a voice assistant. Still, the scenario where Visible and Speak is started first, and then the voice assistant is started, and then the voice assistant exits and resumes Visible and Speak is described as an example. In this scenario, the detailed real-time method of each step of the calling method of the voice recognition engine can be referred to the above Fig.10 and Fig.11 Detailed description in the flowchart.

[0254] When Visible is Talking is started for the first time, VoiceKit is in an unstarted state. In some embodiments, Visible is Talking can first determine whether Visible is Talking has saved the call parameters from the ASR engine. Since Visible is Talking is started for the first time, Visible is Talking has not saved the call parameters from the ASR engine. Therefore, Visible is Talking sends a call request (such as call request 6) for the initialization interface of the ASR engine to VoiceKit, and VoiceKit needs to be initialized first. Then, VoiceKit forwards call request 6 to the ASR engine.

[0255] In response to the call request 6, the ASR engine initializes the service data to the service data corresponding to Visible and Speakable, and provides services for Visible and Speakable. At the same time, the ASR engine generates a call parameter corresponding to Visible and Speakable, and returns the call parameter corresponding to Visible and Speakable to Visible and Speakable for storage. The ASR engine saves the current call parameter as the call parameter corresponding to Visible and Speakable.

[0256] Afterwards, when the voice assistant needs to be started, as the second voice control function, the voice assistant can also first determine whether the voicekit is in the startup state. As can be seen before, the voicekit has been started during the startup process. Therefore, if it is, the voice assistant can directly send a call request for the initialization interface to the ASR engine through the voicekit (such as call request 7).

[0257] In response to the call request 7, the ASR engine will cancel the service data corresponding to the visible and audible voice, and then initialize the service data to the service data corresponding to the voice assistant. In addition, the ASR engine will generate call parameters corresponding to the voice assistant, and return the call parameters corresponding to the voice assistant to the voice assistant for storage. At the same time, the ASR engine will update the current call parameters to the call parameters corresponding to the voice assistant.

[0258] After that, when the voice assistant exits, it will send a call request for the deregistration interface to the ASR engine through voicekit (such as call request 8), so that the ASR engine can deregister the service data corresponding to the voice assistant. At the same time, the voice assistant triggers a recording status change event.

[0259] After Visible and Talk detects the recording status change event, it can restore the voice control of Visible and Talk. Visible and Talk can first check whether VoiceKit is in the startup state. From the previous steps, we can know that when Visible and Talk is restored, VoiceKit is already in the startup state. Therefore, when Visible and Talk restores voice control, there is no need to re-initialize VoiceKit.

[0260] In addition, Visible and Talkable can determine whether Visible and Talkable has saved the call parameters from the ASR engine. From the above steps, it can be seen that after Visible and Talkable is started for the first time, it receives and saves the call parameters corresponding to Visible and Talkable returned by the ASR engine. Therefore, when Visible and Talkable resumes voice control, it can be determined that Visible and Talkable has saved the call parameters from the ASR engine. At this time, Visible and Talkable can query whether the current call parameters of the ASR engine are consistent with the call parameters corresponding to Visible and Talkable.

[0261] Since Visible is Speakable initializes in response to call request 7 and updates the current call parameters to the call parameters corresponding to the voice assistant when providing services to the voice assistant. Therefore, when Visible is Speakable resumes voice control, it will find that the current call parameters of the ASR engine are inconsistent with the call parameters corresponding to Visible is Speakable. At this time, Visible is Speakable needs to reinitialize the ASR engine. VoiceKit is in the startup state, and Visible is Speakable sends a call request for the initialization interface to the ASR engine through VoiceKit (such as call request 9). Afterwards, after the service data corresponding to the voice assistant is deregistered, the ASR engine can respond to the call request 9 and reinitialize the service data to the service data corresponding to Visible is Speakable.

[0262] It can be seen from the description of the above embodiments that in the scenario where Visible and Talk is started first and the voice assistant is started later, and the voice assistant exits and resumes Visible and Talk, the ASR engine switches from Visible and Talk to the voice assistant, and the ASR engine switches from the voice assistant to Visible and Talk again, all involve the voice control function sending a call request for the initialization interface directly to the ASR engine through the voicekit when the voicekit is in the started state. In this way, when the ASR engine switches services, the ASR engine can achieve the effect of self-registration and reinitialization. This avoids multiple destruction and initialization of the voicekit when the ASR engine switches services, reduces the delay when switching services, and reduces the waste of functions.

[0263] In other embodiments, the first voice control function may be visible and spoken, and the second voice control function may be a free command. Fig.12 , which shows a flow chart of a method for calling a speech recognition engine in a scenario where Visible and Speak is started first, Free-to-Mind Command is started later, Free-to-Mind Command is exited, and Visible and Speak is restored.

[0264] S601. It can be seen that the voice control trigger event 3 is received.

[0265] In the embodiment of the present application, the first voice control function can be spoken as long as it is visible, and therefore, the above-mentioned voice control trigger event 3 is the first voice control trigger event.

[0266] S602. In response to voice control triggering event 3, start the see-and-say voice control function.

[0267] S603. It can be seen that the call request 10 is sent to the initialization interface of the ASR engine through voicekit.

[0268] It can be seen that in response to the above voice control trigger event, it will first determine whether VoiceKit is in the startup state. Fig.12 In the scenario shown, before S601, the mobile phone has not started the voicekit. In other words, it can be seen that when the voice control trigger event 3 is received, the voicekit is in an inactive state. Therefore, it can be seen that after sending the call request 10 for the initialization interface of the ASR engine to the voicekit in response to the voice control trigger event 3, the voicekit needs to be initialized first, such as S604.

[0269] S604. In response to the call request 10, voicekit is initialized.

[0270] S605. Voicekit sends a call request 10 for the initialization interface of the ASR engine to the ASR engine.

[0271] S606. The ASR engine binds to the voicekit in response to the call request 10.

[0272] S607. The ASR engine initializes the service data to be visible, that is, the corresponding service data 3.

[0273] In the embodiment of the present application, the first voice control function is visible and audible, so the above service data 3 is the first service data.

[0274] S608. The mobile phone receives voice control trigger event 4.

[0275] In some embodiments, the voice control trigger event 4 may be a preset scene trigger event. In some embodiments, the voice control function of the free command is triggered in scenes such as the alarm ringing, the incoming call ringing, and the answering of the call. Therefore, when the scene triggers such as the alarm ringing, the incoming call ringing, and the answering of the call are detected, the voice control function of the free command is activated.

[0276] The voice control function of the free command needs to call the ASR engine to implement. In some embodiments, the free command can be directly connected to the ASR engine.

[0277] S609. In response to voice control trigger event 4, start the voice control function of the free command.

[0278] S610. The command sends a call request 11 for the initialization interface of the ASR engine to the ASR engine.

[0279] S611. The ASR engine responds to the call request 11, cancels the service data 3 corresponding to the visible command, and initializes the service data to the service data 4 corresponding to the free command.

[0280] S612. In response to the second exit event, the Sui Xin command exits and sends a call request 12 for the logout interface of the ASR engine to the ASR engine.

[0281] S613. The ASR engine responds to the call request 12 and cancels the service data 4 corresponding to the Sui Xin instruction.

[0282] S614. The recording status change event is triggered by the command.

[0283] S615. It can be seen that in response to the recording state change event, a call request 13 for the initialization interface of the ASR engine is sent to the ASR engine through the voicekit.

[0284] S616. The ASR engine determines whether the current engine state is an idle state.

[0285] S617. In response to the call request 13, the ASR engine initializes the service data to be visible, that is, the corresponding service data 3.

[0286] In the embodiment of the present application, Visible and Talk is started first, and then the Free-to-Have Command is started and then exited. Therefore, after the Free-to-Have Command is exited, the voice control function based on Visible and Talk can be restored. In addition, the first voice control function is Visible and Talk, so the above recording state change event also belongs to the first voice control triggering event.

[0287] In the process of starting and calling the ASR engine in S601, voicekit has been initialized. In the process of starting the free command, the free command directly sends a call request 11 to the ASR engine, and the ASR engine can cancel the service data 3 of Visible and Talkable in response to the call request 11, and initialize the service data to the service data 4 corresponding to the free command. In other words, the ASR engine can complete the cancellation and reload process by itself according to the call request 11, without canceling the voicekit. Therefore, when Visible and Talkable calls the ASR engine again in response to the recording status change event, the voicekit is still in the startup state. Therefore, Visible and Talkable can directly send a call request 13 to the ASR engine through voicekit. Therefore, in the process of the ASR engine switching different services, the problem of repeated cancellation and reinitialization of voicekit can be avoided, the delay during service switching can be reduced, and the waste of power consumption can be reduced.

[0288] in addition, Fig.13 The overall process of calling the above-mentioned speech recognition engine is shown. Fig.13APP1 and APP2 shown in the figure can correspond to any voice control function that needs to call the ASR engine to implement the corresponding function. For example, the voice control function can be see and speak, voice assistant, and free instructions, etc. In this embodiment, the calling method of the voice recognition engine includes steps 1 to 5:

[0289] Step 1: APP2 queries the application status queue in the ASR engine by caching the ASR engine handle handler and the corresponding ID. If the handler saved in the current application status queue corresponds to APP1, it means that the current user of the ASR engine is already APP2, and there is no need to switch initialization. No operation is required at this time.

[0290] If the current application status queue does not contain APP2 related information, it is considered that the current ASR engine user is not APP2. At this time, the id in the application status queue is used to query whether the application APP1 is in the set switchable whitelist. If APP1 is not in the switchable whitelist, the switch failure is returned to APP2, and the current APP2 has no permission to switch the use of the ASR engine. If APP1 is in the switchable whitelist, go to step 2.

[0291] Step 2: Check whether the engine state used by APP1 currently occupying the ASR engine is in the deregistering state. If it is in the deregistering state, go to step 3; if it is not in the deregistering state, go to step 4.

[0292] Step 3: After waiting for APP1 to deregister the ASR engine, delete the relevant information of APP1 in the application status queue. Then, insert the application information of APP2 that currently needs to use the ASR engine into the head of the application status queue. At the same time, the ASR engine initializes the service data to the service data corresponding to APP2 and starts normal use. Then, you can choose to go to step 5.

[0293] Step 4: Unregister the ASR engine resources currently being used by APP1, and reinitialize the service data as the ASR engine for use by APP2. At the same time, insert the relevant information of APP2 into the head of the current application status queue, generate and store the handler corresponding to APP2. Then, you can optionally proceed to step 6.

[0294] Step 5: When APP2 calls the deregistration interface of the ASR engine, the engine resources used by APP2 are deregistered, and APP2's call to the ASR engine ends.

[0295] Step 6: When APP2 calls the ASR engine deregistration interface, the related engine resources are deregistered and the ASR engine is reinitialized. At the same time, the relevant status information of APP2 at the head of the application status queue is deleted and the status is restored based on the relevant status information of APP2 cached in the status queue to provide it to the suspended APP1 for continued use.

[0296] It can be seen from the description of the above embodiments that Fig.13 In the process shown, in step 4, the engine resources used by APP1 are deregistered, and the ASR engine is reinitialized for use by APP2, which actually suspends APP1. In some embodiments, when APP1 is suspended and APP2 uses the ASR engine, the suspended APP1 may call the ASR engine related interface for some reason (such as a test or process end trigger).

[0297] In this case, if APP1 calls the ASR engine's deregistration (destroy) interface, the relevant status information of the suspended APP1 in the application status queue is deleted to ensure the normal operation of the ASR engine. In this way, when APP2 ends the call to the ASR engine in the above step 6, the mobile phone will no longer restore APP1's call to the ASR engine based on the relevant information of APP1 in the application status queue. This ensures that the ASR engine functions normally.

[0298] If the suspended APP1 calls a non-destroy interface of the ASR engine, the ASR engine can ignore it, so that APP2 can use the ASR engine to implement the corresponding voice control function normally.

[0299] Some other embodiments of the present application provide an electronic device, which may be the above-mentioned mobile phone. The electronic device may include: a microphone, a memory and one or more processors. The recording device and the memory are coupled to the processor respectively. The recording device is used to collect audio data. The memory is also used to store computer program code, and the computer program code includes computer instructions. When the processor executes the computer instructions, the electronic device can execute the various functions or steps executed by the mobile phone in the above-mentioned method embodiment. In some embodiments, the above-mentioned electronic device may also include a display screen, which is used to display the interface of the electronic device, and information such as prompt icons may also be displayed in the displayed interface. The structure of the electronic device can refer to Figure 7 The structure of the electronic device 100 is shown.

[0300] The present application also provides a chip system, such as Fig.14As shown, the chip system 140 includes at least one processor 1401 and at least one interface circuit 1402. The processor 1401 and the interface circuit 1402 can be interconnected via lines. For example, the interface circuit 1402 can be used to receive signals from other devices (such as a memory of an electronic device). For another example, the interface circuit 1402 can be used to send signals to other devices (such as the processor 1401). Exemplarily, the interface circuit 1402 can read instructions stored in the memory and send the instructions to the processor 1401. When the instruction is executed by the processor 1401, the electronic device can execute the various steps in the above embodiments. Of course, the chip system may also include other discrete devices, which are not specifically limited in the embodiments of the present application.

[0301] An embodiment of the present application also provides a computer-readable storage medium, which includes computer instructions. When the computer instructions are executed on the above-mentioned electronic device (such as a mobile phone), the electronic device executes each function or step executed by the mobile phone in the above-mentioned method embodiment.

[0302] The present application also provides a computer program product, which, when executed on a computer, enables the computer to execute the functions or steps executed by the mobile phone in the above method embodiment. The computer may be an electronic device, such as a mobile phone.

[0303] Through the description of the above implementation methods, technical personnel in the relevant field can clearly understand that for the convenience and simplicity of description, only the division of the above-mentioned functional modules is used as an example. In actual applications, the above-mentioned functions can be assigned to different functional modules as needed, that is, the internal structure of the device can be divided into different functional modules to complete all or part of the functions described above.

[0304] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic, for example, the division of modules or units is only a logical function division, and there may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another device, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0305] The units described as separate components may or may not be physically separated, and the components shown as units may be one physical unit or multiple physical units, that is, they may be located in one place or distributed in multiple different places. Some or all of the units may be selected according to actual needs to achieve the purpose of the present embodiment.

[0306] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0307] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a readable storage medium. Based on this understanding, the technical solution of the embodiment of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, which is stored in a storage medium, including several instructions to enable a device (which can be a single-chip microcomputer, chip, etc.) or a processor (processor) to perform all or part of the steps of the various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read only memory (ROM), random access memory (RAM), disk or optical disk and other media that can store program code.

[0308] The above contents are only specific implementation methods of the present application, but the protection scope of the present application is not limited thereto. Any changes or substitutions within the technical scope disclosed in the present application shall be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.

Claims

1. A method for calling a speech recognition engine, characterized in that: The method is applied to an electronic device, the electronic device having a first voice control function; the electronic device comprising a voice recognition ASR engine and a voice kit voicekit; a first voice control module corresponding to the first voice control function is connected to the ASR engine through the voicekit; the method comprises: The first voice control module receives a first voice control trigger event; When the voicekit is in the startup state, the first voice control module sends a first call request to the initialization interface of the ASR engine through the voicekit in response to the first voice control trigger event; In response to the first calling request, the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module to provide service for the first voice control function.

2. The method according to claim 1, characterized in that When the voicekit is in the startup state, the first voice control module sends a first call request to the initialization interface of the ASR engine through the voicekit in response to the first voice control trigger event, including: If the current user of the ASR engine is not the first voice control module, then when the voicekit is in the started state, the first voice control module responds to the first voice control trigger event and sends a first call request to the initialization interface of the ASR engine through the voicekit.

3. The method according to claim 2, characterized in that After the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module in response to the first call request, the method further includes: The ASR engine generates a first calling parameter corresponding to the first voice control module, and saves the first calling parameter as a current calling parameter of the ASR engine; The ASR engine sends the first calling parameter to the first voice control module; The first voice control module receives and saves the first calling parameter.

4. The method according to claim 3, characterized in that The current user of the ASR engine is not the first voice control module, comprising: When receiving the first voice control trigger event, the first voice control module does not store the calling parameters from the ASR engine; The first voice control triggering event includes a first starting event of the first voice control function.

5. The method according to claim 4, characterized in that The ASR engine, in response to the first call request, initializes the service data of the ASR engine to the first service data corresponding to the first voice control module, including: The ASR engine cancels the service data corresponding to the current user in response to the first call request; The ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module.

6. The method according to claim 3, characterized in that The current user of the ASR engine is not the first voice control module, comprising: When the first voice control trigger event is received, the first voice control module stores the calling parameters from the ASR engine, and the calling parameters stored by the first voice control module are inconsistent with the current calling parameters of the ASR engine; The first voice control triggering event includes: a recording state change event triggered when the current user of the ASR engine exits.

7. The method according to claim 6, characterized in that The electronic device also has a second voice control function; the current user of the ASR engine is a second voice control module corresponding to the second voice control function, and the current calling parameter of the ASR engine is a second calling parameter corresponding to the second voice control function; Before the first voice control module receives the first voice control triggering event, the method further includes: The second voice control module receives a second start event of a second voice control function; The second voice control module sends a second call request to the initialization interface of the ASR engine through the voicekit in response to the second start event; In response to the second calling request, initializing the service data of the ASR engine to second service data corresponding to the second voice control function; The second voice control module receives an exit event of the second voice control function; The second voice control module triggers a recording state change event in response to the exit event, and sends a third call request to the logout interface of the ASR engine through the voicekit; The ASR engine cancels the second service data in response to the third call request; The step of saving the first calling parameter as the current calling parameter of the ASR engine includes: the ASR engine updating the current calling parameter from the second calling parameter to the first calling parameter.

8. The method according to any one of claims 2 to 7, characterized in that: The ASR engine, in response to the first call request, initializes the service data of the ASR engine to the first service data corresponding to the first voice control module, including: When it is determined that the ASR engine meets the preset switching conditions, the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module in response to the first call request; the preset switching conditions are used to indicate that the ASR engine allows the current user to switch to the first voice control module.

9. The method according to any one of claims 2 to 8, characterized in that: The method further comprises: If the current user of the ASR engine is the first voice control module, the first voice control module uses the ASR engine to perform voice control via the voicekit.

10. The method according to any one of claims 1 to 9, characterized in that After the first voice control module receives the first voice control triggering event, the method further includes: When the voicekit is in an inactive state, the first voice control module sends a first call request to the voicekit in response to the first voice control triggering event; The voicekit is started in response to the first call request; The voicekit sends the first call request to the ASR engine; In response to the first calling request, the ASR engine initializes the service data of the ASR engine to the first service data corresponding to the first voice control module.

11. An electronic device, characterized in that: The electronic device comprises: a processor, a memory and a microphone; the memory and the microphone are respectively coupled to the processor; The microphone is used to collect audio data; the memory stores computer program code, and the computer program code includes computer instructions. When the computer instructions are executed by the processor, the electronic device executes the method as described in any one of claims 1 to 10.

12. A computer-readable storage medium, characterized in that: The method comprises computer instructions, and when the computer instructions are executed on an electronic device, the electronic device executes the method as claimed in any one of claims 1 to 10.

Images