Voice data processing method and device, electronic equipment and storage medium

By querying the business status of registered applications in the breath wake-up engine application and distributing voice data processing tasks, the problem of only setting up applications in the prior art is solved, and the flexibility and user experience of awakening different applications are improved.

CN119028332AActive Publication Date: 2024-11-26HONOR DEVICE CO LTD

Patent Information

Application Number
CN202310598073.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2023-05-24
Publication Date
2024-11-26
Estimated Expiration
2043-05-24

Smart Images

  • Figure CN119028332A_ABST
    Figure CN119028332A_ABST
Patent Text Reader

Abstract

The invention provides a voice data processing method and device, electronic equipment and a storage medium, and is applied to the technical field of information. Under the condition that a breath wake-up event is detected by a breath wake-up engine application, the breath wake-up engine application obtains input voice data, and queries the service state of each registered application which completes registration to the breath wake-up engine application; the breath wake-up engine application determines a target application in the registered applications according to the service states of the registered applications; the breath wake-up engine application distributes the processing task of the voice data to the target application; and the target application obtains and processes the voice data according to the processing task. By applying the method provided by the invention, the target application can be determined from the registered applications by querying the service states of the registered applications under the condition that the breath wake-up event is detected, so that different application programs can be awakened.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] The present application relates to the field of information technology, and in particular to a voice data processing method, device, electronic device and storage medium. Background Art

[0002] With the development of voice recognition technology, voice assistants have been added to many electronic devices to help users complete the human-computer interaction process with electronic devices. Generally speaking, the voice assistant is in a dormant state. When the user wants to use the voice assistant, the voice assistant can be awakened by two technologies: button awakening and keyword awakening. However, in the above implementation process, waking up the voice assistant is relatively cumbersome, and breath awakening came into being.

[0003] However, in the related art, each application that can be awakened is mutually exclusive, and the user can only wake up a certain application. For example, if the user turns on the breath awakening switch of the voice assistant, the breath awakening switches of other applications will be turned off. The user can only wake up the voice assistant application when waking up, which is not flexible enough. Summary of the invention

[0004] The present application provides a voice data processing method, device, electronic device and storage medium to improve the flexibility of breath awakening applications.

[0005] In a first aspect of the present application, a voice data processing method is provided, which is applied to an electronic device, wherein a breath wake-up engine application is running in the electronic device, and the method comprises:

[0006] When the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application;

[0007] The breath awakening engine application determines a target application from among the registered applications according to the service status of each of the registered applications;

[0008] The breath awakening engine application distributes the processing task of the voice data to the target application;

[0009] The target application acquires and processes the voice data according to the processing task.

[0010] In a possible implementation, the method further includes:

[0011] When the breath wake-up engine application detects access to at least one registered application, the breath wake-up path is opened to detect a breath wake-up event; when the breath wake-up engine application does not detect access to a registered application, the breath wake-up path is closed.

[0012] In a possible implementation, the method further includes:

[0013] After the application to be registered is started, the application to be registered runs the breath wake-up switch and maintains the state of its own breath wake-up switch, and sends a switch registration message to the breath wake-up engine application;

[0014] The breath wake-up engine registers the breath wake-up switch of the application to be registered according to the switch registration message; wherein, the application to be registered after the breath wake-up switch is successfully registered is the registered application.

[0015] In a possible implementation, the breath awakening engine application determines the target application in each of the registered applications according to the service status of each of the registered applications, including:

[0016] The breath awakening engine application selects the registered application with focus from the registered applications running in the foreground to obtain the target application according to the business status of each of the registered applications.

[0017] In a possible implementation, the method further includes:

[0018] The breath awakening engine application determines the voice assistant application as the target application based on the business status of each registered application when it is detected that there is no registered application running in the foreground or the focus is not on the registered application.

[0019] In a possible implementation, the registered applications include an input method application and a voice assistant application;

[0020] The breath awakening engine application determines a target application from among the registered applications according to the service status of each of the registered applications, including:

[0021] The breath wake-up engine application determines that the input method application is the target application based on the business status of each registered application and detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application; otherwise, the voice assistant application is determined to be the target application.

[0022] In a second aspect of the implementation of the present application, a voice data processing device is provided, which is applied to an electronic device, wherein a breath awakening engine application is running in the electronic device, and the device includes:

[0023] A service status query module, used for, when the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application;

[0024] A target application determination module, used for the breath awakening engine application to determine a target application among the registered applications according to the business status of each registered application;

[0025] A task distribution module, used for the breath awakening engine application to distribute the processing task of the voice data to the target application;

[0026] The voice data processing module is used for the target application to obtain and process the voice data according to the processing task.

[0027] In a possible implementation, the device further includes:

[0028] The breath awakening path management module is used to open the breath awakening path to detect the breath awakening event when the breath awakening engine application detects the access of at least one registered application; and close the breath awakening path when the breath awakening engine application does not detect the access of the registered application.

[0029] In a possible implementation, the device further includes:

[0030] A registration information sending module, used for, after the application to be registered is started, the application to be registered runs the breath wake-up switch and maintains the state of its own breath wake-up switch, and sends a switch registration message to the breath wake-up engine application;

[0031] The application registration module is used for the breath wake-up engine to register the breath wake-up switch of the application to be registered according to the switch registration message; wherein, the application to be registered after the breath wake-up switch is successfully registered is the registered application.

[0032] In a possible implementation, the target application determination module includes:

[0033] The target application determination submodule is specifically used for the breath awakening engine application to select the registered application with focus from the registered applications running in the foreground to obtain the target application according to the business status of each registered application.

[0034] In a possible implementation, the device further includes:

[0035] The registered application status query module is used for the breath awakening engine application to determine the voice assistant application as the target application according to the business status of each registered application when it is detected that there is no registered application running in the foreground or the focus is not on the registered application.

[0036] In a possible implementation, the registered applications include an input method application and a voice assistant application;

[0037] The target application determination module includes:

[0038] The input method application status query module, the breath wake-up engine application determines that the input method application is the target application according to the business status of each registered application, when it detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application; otherwise, the voice assistant application is determined to be the target application.

[0039] According to a third aspect of the present application, an electronic device is provided, including:

[0040] at least one processor; and

[0041] a memory communicatively connected to the at least one processor; wherein,

[0042] The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor so that the at least one processor can perform the voice data processing method described in any one of the present application.

[0043] Another aspect of the present application provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to enable the computer to execute any of the speech data processing methods described in the present application.

[0044] Another aspect of the present application provides a computer program product, including a computer program, which implements the speech data processing method described in any one of the present application when executed by a processor.

[0045] Beneficial effects of the embodiments of the present application:

[0046] By applying the method of the present application, when a breath wake-up event is detected, the target application can be determined from the registered applications by querying the business status of the registered applications, thereby waking up different application programs and improving the flexibility of the breath wake-up application.

[0047] Of course, implementing any product or method of the present application does not necessarily require achieving all of the advantages described above at the same time. BRIEF DESCRIPTION OF THE DRAWINGS

[0048] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application, and for ordinary technicians in this field, other embodiments can also be obtained based on these drawings.

[0049] Figure 1 A schematic diagram of a breath awakening event provided in an embodiment of the present application;

[0050] Figure 2 A schematic diagram of a first structure of an electronic device provided in an embodiment of the present application;

[0051] Figure 3 A second structural diagram of an electronic device provided in an embodiment of the present application;

[0052] Figure 4 A software structure block diagram of an electronic device provided in an embodiment of the present application;

[0053] Figure 5 A flow chart of a method for processing voice data provided in an embodiment of the present application;

[0054] Figure 6 A schematic diagram of voice data processing provided in an embodiment of the present application;

[0055] Figure 7 A schematic diagram of a breath wake-up switch provided in an embodiment of the present application;

[0056] Figure 8 A schematic diagram of the running state of the input method application provided in the embodiment of the present application;

[0057] Fig. 9 A schematic diagram of operating a mobile phone with one hand provided in an embodiment of the present application;

[0058] Fig.10 A schematic diagram of a user performing a breath awakening operation provided in an embodiment of the present application;

[0059] Fig.11 A schematic diagram of the structure of a voice data processing device provided in an embodiment of the present application. DETAILED DESCRIPTION

[0060] The following will be combined with the drawings in the embodiments of the present application to clearly and completely describe the technical solutions in the embodiments of the present application. Obviously, the described embodiments are only part of the embodiments of the present application, not all of the embodiments. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in this field based on the present application belong to the scope of protection of the present application.

[0061] In the related art, there are two ways to wake up an application: one is to wake up the application with a keyword, and the other is to wake up the application with a breath. For a keyword to wake up an application (application, APP):

[0062] After the electronic device collects voice data through a sound collection sensor (for example, a microphone), it sends it to a low-power audio digital signal processor (ADSP), and the ADSP detects the voice data through a voice wake-up model.

[0063] When ADSP detects that there is wake-up word data matching the preset wake-up word in the voice data, ADSP sends the voice data to the wake-up software module. The wake-up software module performs text verification and voiceprint verification on the aforementioned voice data to determine whether the voice data matches the generated voiceprint model. When the voice data matches the voiceprint model, the wake-up software module can control the startup of the application. Among them, the keyword wake-up application has specific keywords. After each wake-up, the voice wake-up model can stop running once and start running once.

[0064] For the Breath Wake-up App:

[0065] After the electronic device collects voice data through the sound collection sensor and gesture data through the inertial detection sensor, it sends them to a low-power audio digital signal processor (ADSP). The ADSP detects the voice data and gesture data through the breath awakening model.

[0066] Among them, the breath awakening model is used to detect the acquired voice data and gesture data, and to determine whether the similarity between the voice data and the preset awakening breath data is greater than a first threshold, and whether the similarity between the gesture data and the preset awakening gesture data is greater than a second threshold.

[0067] In addition, the original awakening model can be trained using sample awakening breath data and sample awakening gesture data to obtain a breath awakening model.

[0068] When the ADSP detects that the similarity between the voice data and the preset awakening breath data is greater than a first threshold, and the similarity between the gesture data and the preset awakening gesture data is greater than a second threshold, the ADSP sends the voice data to the breath awakening software module. The breath awakening software module controls the application to start.

[0069] However, whether it is keyword awakening or breath awakening, it can only be used to awaken a pre-set application, and the application awakening method is not flexible enough.

[0070] The present application provides a voice data processing method, device, electronic device and storage medium. When a breath wake-up engine application detects a breath wake-up event, the breath wake-up engine application obtains input voice data and queries the business status of each registered application that has completed registration with the breath wake-up engine application. The breath wake-up engine application determines the target application among the registered applications according to the business status of each registered application. The breath wake-up engine application distributes the processing task of the voice data to the target application. The target application obtains the voice data and processes it according to the processing task.

[0071] By applying the method of the present application, when a breath wake-up event is detected, the target application can be determined from the registered applications by querying the business status of the registered applications, thereby waking up different application programs and improving the flexibility of the breath wake-up application.

[0072] The method for waking up an application provided in the present application can be applied to an electronic device, which can be an electronic device with display hardware and corresponding software support.

[0073] For example, the electronic device may be a mobile phone, a tablet computer, a wearable device, a vehicle-mounted device, a laptop computer, an ultra-mobile personal computer (UMPC), a netbook, a personal digital assistant (PDA), a household appliance, or the like. The present application does not impose any restrictions on the specific type of the electronic device.

[0074] like Figure 1 As shown, Figure 1 A schematic diagram of a scenario of a method for waking up an application provided by an embodiment of the present application is shown, in which a user can lift up an electronic device, bring the bottom of the electronic device close to the mouth, and speak into the microphone.

[0075] After the electronic device collects the corresponding voice data when the user speaks through the microphone, and the corresponding gesture data when the user lifts the electronic device through the inertial detection sensor, it is sent to the low-power ADSP, and the ADSP detects the voice data and gesture data through the breath awakening model. When the ADSP detects that the similarity between the voice data and the preset awakening breath data is greater than the first threshold, and the similarity between the gesture data and the preset awakening gesture data is greater than the second threshold, the ADSP sends the voice data to the breath awakening software module. The breath awakening software module controls the startup of the application.

[0076] Among them, the distance between the electronic device and the user's mouth can be maintained at 0-5cm, which makes it easier for the microphone of the electronic device to accurately collect the user's voice data.

[0077] In addition, the above-mentioned gesture data may be wrist-raising gesture data.

[0078] in, Figure 1 From the gesture state indicated by the box A in the left figure to the gesture state indicated by the box B in the right figure, it can be seen that the user can lift the electronic device by raising the wrist, and the inertial detection sensor can collect the gesture data of the wrist raising.

[0079] Figure 1 As can be seen from the right picture, after the user raises the electronic device with the wrist-raising gesture, he or she puts his or her mouth close to the microphone of the electronic device to speak, and can emit the breath indicated by part C. The microphone can collect the breath and the voice data corresponding to the breath.

[0080] It should be understood that the above is an example of a scenario and does not limit the scenario of the present application.

[0081] First, in a first aspect of an embodiment of the present application, an electronic device is provided, such as Figure 2 As shown, the electronic device includes:

[0082] Memory 201, used for storing computer programs;

[0083] The processor 202 is used to execute the program stored in the memory 201, and implements the following steps:

[0084] When the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains the input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application;

[0085] The breath awakening engine application determines the target application among the registered applications according to the business status of each registered application;

[0086] The breath wake-up engine application distributes the processing task of voice data to the target application;

[0087] The target application obtains and processes the voice data according to the processing task.

[0088] Furthermore, the electronic device may further include a communication bus and / or a communication interface, and the processor 202, the communication interface, and the memory 201 communicate with each other via the communication bus.

[0089] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0090] The communication interface is used for communication between the above electronic device and other devices.

[0091] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0092] The above-mentioned processor can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0093] For ease of explanation, a mobile phone is used as an example of an electronic device.

[0094] like Figure 3 As shown, in some embodiments, the electronic device 300 may include a processor 301 and a communication module 302 .

[0095] The processor 301 and Figure 2The processor 202 may be the same as the processor 201, and may include one or more processing units. For example, the processor 301 may include an application processor (AP), a modem processor, a graphics processor, an image signal processor (ISP), a controller, a memory, a video stream codec, a digital signal processor, a baseband processor, and / or a neural-network processing unit (NPU), etc. Different processing units may be independent devices or may be integrated in one or more processors 301.

[0096] The controller may be the nerve center and command center of the electronic device 300. The controller may generate an operation control signal according to the instruction operation code and the timing signal to complete the control of fetching and executing instructions.

[0097] The processor 301 may also be provided with a memory for storing instructions and data.

[0098] In some embodiments, the memory in the processor 301 is a cache memory. The memory can store instructions or data that the processor 301 has just used or cyclically used. If the processor 301 needs to use the instruction or data again, it can be directly called from the memory. This avoids repeated access, reduces the waiting time of the processor 301, and thus improves the efficiency of the system.

[0099] In some embodiments, the processor 301 may include one or more interfaces. The interface may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0100] The communication module 302 may include an antenna 1, an antenna 2, a mobile communication module, and / or a wireless communication module.

[0101] like Figure 3 As shown, in some embodiments, the electronic device 300 may also include an external memory interface 305, an internal memory 304, a USB interface 306, a charging management module 307, a power management module 308, a battery 309, and a sensor module 303, etc.

[0102] NPU is a neural network (NN) computing processor. By drawing on the structure of biological neural networks, such as the transmission mode between neurons in the human brain, it can quickly process input information and can also continuously self-learn. Through NPU, applications such as intelligent cognition of electronic device 300 can be realized, such as image recognition, face recognition, voice recognition, text understanding, etc.

[0103] The charging management module 307 is used to receive charging input from a charger, where the charger can be a wireless charger or a wired charger.

[0104] In some wired charging embodiments, the charging management module 307 may receive charging input from a wired charger through the USB interface 306 .

[0105] In some wireless charging embodiments, the charging management module 307 may receive wireless charging input via a wireless charging coil of the electronic device 300. While the charging management module 307 is charging the battery 309, it may also provide power to the electronic device 300 via the power management module 308.

[0106] The power management module 308 is used to connect the battery 309, the charging management module 307 and the processor 301. The power management module 308 receives input from the battery 309 and / or the charging management module 307, and supplies power to the processor 301, the internal memory 304, the external memory, and the communication module 302. The power management module 308 can also be used to monitor parameters such as battery capacity, battery cycle number, and battery health status (leakage, impedance).

[0107] In some other embodiments, the power management module 308 may also be disposed in the processor 301 .

[0108] In other embodiments, the power management module 308 and the charging management module 307 may also be disposed in the same device.

[0109] The external memory interface 305 can be used to connect an external memory card, such as a Micro SD card, to expand the storage capacity of the electronic device 300. The external memory card communicates with the processor 301 through the external memory interface 305 to implement a data storage function. For example, files such as music and video streams are stored in the external memory card.

[0110] The internal memory 304 can be used to store computer executable program codes, and the executable program codes include instructions. The processor 301 executes various functional applications and data processing of the electronic device 300 by running the instructions stored in the internal memory 304. The internal memory 304 may include a program storage area and a data storage area. Among them, the program storage area can store an operating system, an application required for at least one function (such as a sound playback function, an image playback function, etc.), etc. The data storage area can store data created during the use of the electronic device 300 (such as audio data, a phone book, etc.), etc. In addition, the internal memory 304 may include a high-speed random access memory, and may also include a non-volatile memory, such as at least one disk storage device, a flash memory device, a universal flash storage (UFS), etc.

[0111] The sensor module 303 in the electronic device 300 may include image sensors, touch sensors, pressure sensors, gyroscope sensors, air pressure sensors, magnetic sensors, acceleration sensors, distance sensors, proximity light sensors, ambient light sensors, fingerprint sensors, temperature sensors, bone conduction sensors and other components to realize the sensing and / or acquisition functions of different signals.

[0112] Optionally, the electronic device 300 may further include peripheral devices, such as a mouse, buttons, indicator lights, a keyboard, a speaker, a microphone, etc.

[0113] The buttons include a power button, a volume button, etc. The buttons may be mechanical buttons or touch buttons. The electronic device 300 may receive button inputs and generate key signal inputs related to user settings and function control of the electronic device 300.

[0114] The indicator may be a light indicator, which may be used to indicate the charging status and the change in the power level, and may also be used to indicate messages, missed calls, and notifications, etc.

[0115] It can be understood that the structure illustrated in this embodiment does not constitute a specific limitation on the electronic device 300 .

[0116] In other embodiments, the electronic device 300 may include more or fewer components than shown, or combine some components, or separate some components, or arrange the components differently. The components shown may be implemented in hardware, software, or a combination of software and hardware.

[0117] In electronic devices, the software system can be divided into several layers, such as Figure 4 As shown, Figure 4 A software structure block diagram of an electronic device provided in an embodiment of the present application. The layered architecture divides the software system of the electronic device into several layers, each layer has a clear role and division of labor. The layers communicate with each other through software interfaces. In some embodiments, the electronic device application system can be divided into an application layer (application), a framework layer (framework, fwk), a driver layer (hardware abstract layer, HAL) and a kernel layer (kernel) / chip layer (analog digital singal processor, ADSP).

[0118] The application layer can include a series of application packages, and the application layer runs applications by calling the application programming interface (API) provided by the application framework layer. An application package can include multiple applications, such as a breath wake-up engine application, a voice assistant application, an input method application, a call application, and a map application.

[0119] The framework layer provides API and programming framework for the applications in the application layer. The application framework layer includes some predefined functions. Figure 4 As shown, the application framework layer may include an audio trigger module (Sound Trigger). The audio trigger module may be used to control the start of an application in the application layer.

[0120] In addition, the framework layer may also include a sound trigger module, an audio policy service module, etc. In addition, the application framework layer may also include an audio service module, an audio trigger service module, an audio indication module, etc. The sound trigger module is used to send a wake-up event to the audio trigger module, and to send a notification to the audio driver of the driver layer to instruct the breath wake-up model to stop running or start running. The audio policy service module is used to establish a voice recognition channel with the voice assistant application of the application layer.

[0121] In addition, the audio service module is used to send a startup notification to the audio trigger module in response to the startup notification sent by the application of the application layer. The audio trigger module is also used to send a startup notification to the audio trigger service module in response to the startup notification sent by the audio service module. The audio trigger service module is used to send a startup notification to the sound trigger module in response to the startup notification sent by the audio trigger module. The sound trigger module is also used to send a notification to start running the breath awakening model to the audio strategy service module in response to the startup notification sent by the audio trigger module. The audio strategy service module starts the notification of running the breath awakening model in response to the notification of starting running the breath awakening model. The audio indication module is used to send a notification of loading the breath awakening model to the audio driver of the driver layer 3 in response to the notification of loading the breath awakening model sent by the audio strategy service module. The audio driver of the driver layer is used to send a notification for instructing the start of running the breath awakening model to the audio digital signal processor of the breath awakening processing device.

[0122] The driver layer is a layer between hardware and software, used to drive the hardware to make it work. The driver layer can be installed with multiple drivers for driving the hardware to work. For example, the audio driver (sound trigger hal), the sound trigger driver (Sound Trigger driver) and the audio stream input driver (Audio Stream in driver).

[0123] The kernel layer / chip layer includes ADSP, which is used to determine the presence of a breath wake-up event based on the breath wake-up algorithm. ADSP can obtain the session identifier (identity, ID) corresponding to the wake-up event. The session ID is preset in ADSP by the developer. In an example, after detecting a breath wake-up event, ADSP reports the breath wake-up event including the session ID to the driver layer; the driver layer then reports the breath wake-up event to the framework layer; the framework layer reports the breath wake-up event to the breath wake-up engine application, and the breath wake-up engine application processes the breath wake-up event.

[0124] It should be noted that the application layer, application framework layer, driver layer and kernel layer / chip layer may also include other contents, which are not specifically limited here.

[0125] By applying the method of the embodiment of the present application, the breath wake-up event can be reported layer by layer through the underlying breath wake-up algorithm after the breath wake-up event is identified, so that the breath wake-up engine application can determine the target application for voice data processing according to the business status of the registered application after receiving the breath wake-up event, and it is also beneficial for the target application to obtain voice data for data processing by fetching the stream according to the specified session ID and buffer size (buffer capacity).

[0126] In order to solve the technical problem that only one application can be awakened, the second aspect of the embodiment of the present application provides a voice data processing method applied to an electronic device, in which a breath awakening engine application is running. The method includes: Figure 5 Steps shown:

[0127] Step S501: when the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application.

[0128] Among them, the breath awakening engine application can detect the scene state of the electronic device when detecting the breath awakening event. Take the electronic device as a mobile phone as an example. Figure 1 As shown, when a user issues a voice command to a mobile phone, he often picks up the mobile phone and places it near his mouth for voice input. Therefore, it can be set that after the user lifts the mobile phone through the wrist-raising gesture, the user's mouth is close to the microphone of the mobile phone and speaks as a breath awakening event. Specifically, the breath awakening engine application in the mobile phone can first detect whether the mobile phone is lifted up. After detecting that the mobile phone is lifted up, it can detect whether there is voice input. If so, it is determined that the mobile phone meets the situation of the breath awakening event. When the breath awakening event is detected, the breath awakening engine application will call hardware resources such as microphone and memory to collect voice data. The breath awakening engine application can register multiple applications. After the application is registered, the business status of the registered application can be obtained. The business status of the registered application indicates the working status of the registered application, for example, whether the registered application is a foreground application or a background application, and whether the focus of the user operation is in the registered application; for example, for audio and video software, the audio or video opening can be set as the preset business status; for office software such as spreadsheet documents, the cursor can be set to flash in the file, that is, "input is in progress", as the preset business status. . In one example, when an application is registered, the registration information sent by the application may include the current business status of the application; in other examples, the business status of the application may also be periodically reported by the registered application; or the breath wake-up engine application may notify the registered application to report.

[0129] Step S502: The breath awakening engine application determines a target application among the registered applications according to the business status of the registered applications.

[0130] When a breath awakening event is detected, the breath awakening engine application can query whether the business status of each registered application meets the corresponding preset business status conditions. If so, the registered application that meets the preset business status conditions is determined to be the target application. Among them, each registered application can be an application running in the foreground of the electronic device, or it can be an application running in the background. The application running in the foreground can fully display all the functions of the application and occupy a large amount of system resources. The application running in the background can run lightweight tasks and cannot realize all the functions of the application. In an example, there can only be one target application at the same time. Therefore, when setting the preset business status conditions of each application, it is necessary to ensure that each preset business status condition is exclusive, that is, the preset business status conditions of each application can ensure that there is only one target application at the same time.

[0131] Step S503: The breath awakening engine application distributes the processing task of the voice data to the target application.

[0132] After determining the target application, the breath awakening engine application can wake up the target application and distribute the voice data processing task to the target application. The processing task may include parameters such as the audio session ID of the voice data.

[0133] Step S504: The target application obtains and processes the voice data according to the processing task.

[0134] After receiving the voice data processing task, the target application can obtain the collected voice data according to the processing task, and then process the voice data according to the obtained voice data. For example, the target application can obtain the voice data by fetching the stream through the audio session identifier in the processing task. After obtaining the voice data, the target application can process the voice data based on its own settings, for example, converting the voice data into text data.

[0135] By applying the method of the embodiment of the present application, when a breath wake-up event is detected, the target application can be determined from the registered applications by querying the business status of the registered applications, thereby waking up different application programs and improving the flexibility of the breath wake-up application.

[0136] After registering the application into the breath awakening engine application, whether to open the breath awakening path can be determined by querying whether the registered application is connected to the breath awakening engine application. Specifically, the method of the embodiment of the present application may further include the following steps:

[0137] When the breath wake-up engine application detects access to at least one registered application, the breath wake-up path is opened to detect a breath wake-up event; when the breath wake-up engine application does not detect access to a registered application, the breath wake-up path is closed.

[0138] In actual applications, each registered application can have an independent breath wake-up switch, and each breath wake-up switch does not affect each other. The user can choose whether to turn on the breath wake-up switch in the page operation corresponding to the registered application. When the breath wake-up engine application detects that the state of the breath wake-up switch corresponding to a registered application is turned on, the registered application is considered to be connected to the breath wake-up engine application.

[0139] The breath awakening engine application can have multiple registered applications, and each registered application corresponds to a breath awakening switch. The breath awakening engine application determines whether to open or close the breath awakening path by identifying the state of the switch. If at least one switch is turned on, the breath awakening path is opened. Specifically, the breath awakening engine application sends a breath awakening path opening instruction to the application framework layer, and the application framework layer sends the breath awakening path opening instruction to the driver layer, and then the driver layer sends the breath awakening path opening instruction to the chip layer, so that the chip layer can detect the breath awakening event by running the breath awakening algorithm. The process of closing the breath awakening path is similar to that of opening the breath awakening path, and will not be repeated here. It can be understood that if the breath awakening path is already opened, there is no need to execute the process of opening the breath awakening path; if the breath awakening path is already closed, there is no need to execute the process of closing the breath awakening path.

[0140] By applying the method of the embodiment of the present application, it is possible to detect the access status of registered applications in advance to determine whether to open the breath wake-up path, and then close the breath wake-up path when no registered application is accessed. This can reduce false detection of breath wake-up events and reduce the power consumption of electronic devices.

[0141] Specifically, when registering the application to the breath awakening engine application, the registration can be achieved through the following steps:

[0142] Step 1: After the application to be registered is started, the application to be registered runs the breath wake-up switch and maintains the status of its own breath wake-up switch, and sends a switch registration message to the breath wake-up engine application.

[0143] Among them, the switch registration message sent may include basic information of the application to be registered, such as the identifier (ID), name, storage location information, etc. of the application to be registered. In one example, in order to facilitate the judgment of the business status of the application to be registered, when sending the switch registration message to the breath wake-up engine application, the preset business status condition may also be sent to the breath wake-up engine application, so that the breath wake-up engine application can judge whether the business status of the registered application meets the preset business status condition when the breath wake-up event is detected. In other embodiments, the breath wake-up engine application may also pre-maintain preset business status conditions for multiple applications, all within the protection scope of this application.

[0144] Step 2: The breath wake-up engine registers the breath wake-up switch of the application to be registered according to the switch registration message; wherein, the application to be registered after the breath wake-up switch is successfully registered is the registered application.

[0145] After the breath wake-up switch is successfully registered, the registered application can determine whether to access the breath wake-up engine application by selecting the switch. When the breath wake-up engine application detects that the state of any breath wake-up switch is turned on, it is considered that the registered application with the breath wake-up switch turned on is accessed.

[0146] like Figure 6 As shown, taking the applications to be registered as voice assistant applications and input method applications as examples, the breath wake-up engine application can include a switch status management module and a scene perception decision module. Among them, the scene perception decision module is used to query the business status of each registered application and distribute voice data processing tasks. The switch status management module is used to query the switch status of the registered application, and determine whether to open or close the breath wake-up path according to the switch status. The voice assistant application and the input method application first send the switch registration information to the switch management module in the breath wake-up engine application for switch registration. After the registration is completed, the switch status management module can query the switch status of the registered application. If it is identified that at least one switch of the registered application is turned on, the breath wake-up path is turned on.

[0147] Specifically, the breath wake-up engine maintains a breath wake-up switch list. When a registered application turns on the breath wake-up switch, the breath wake-up switch corresponding to the registered application in the switch management module of the breath wake-up engine is turned on. The user can choose whether to turn on the breath wake-up switch for a certain application in the page operation, such as Figure 7 As shown, Figure 7This is a schematic diagram of the breath wake-up switch of the voice assistant application YOYO. When the breath wake-up switch is turned on for the first time, the switch registration needs to be performed; after successful registration, the breath wake-up switch is turned off to prevent the breath wake-up engine application from waking up the voice assistant application; when there are multiple registered applications, such as application A and application B, the switch management module identifies the status of the breath wake-up switches corresponding to application A and application B respectively. If both are in the off state, the breath wake-up path is closed; if at least one of the breath wake-up switches of the two is in the on state, the breath wake-up path is opened. In an example, when the breath wake-up switches of all registered applications except the voice assistant application are turned off, when the breath wake-up switch of the voice assistant application is changed from on to off, the breath wake-up engine will close the breath wake-up path; when the breath wake-up switch of the voice assistant application is changed from off to on, the breath wake-up engine will open the breath wake-up path, and when the breath wake-up engine application detects a breath wake-up event, it will wake up the voice assistant.

[0148] By applying the method of the embodiment of the present application, the state of the breath wake-up switch can be judged to determine whether the application is registered for the switch. After successful registration, the switch state can be used to judge the access status of the registered application, thereby determining whether to open the breath wake-up path. Further, after detecting the breath wake-up event, the voice data processing task can be sent to the registered application that is connected to the breath wake-up engine application.

[0149] In the above step S502, determining the target application may be determining the registered application with the focus as the target application. Specifically, the breath awakening engine application may select the registered application with the focus from the registered applications running in the foreground to obtain the target application according to the business status of each registered application.

[0150] The focus is an identifier that can indicate the current running state. For example, in a mobile device, the focus is the cursor in the input method application, or the position where the finger clicks; in a computer, the focus can be the mouse, or the position where the finger clicks. In an example, the electronic device is a tablet computer, and the registered application is a text editing software application. When it is identified that the text editing software application is running in the foreground and the mouse is in the text editing software application, the text editing software application can be determined as the target application.

[0151] By applying the method of the embodiment of the present application, the registered application with focus in the foreground can be directly determined as the target application, thereby completing the distribution of voice processing tasks to the target application and the processing of voice data by the target application.

[0152] If the registered application where the focus is selected is not detected in the registered applications running in the foreground in the above embodiment, the method of the embodiment of the present application may further include:

[0153] The breath wake-up engine application determines the voice assistant application as the target application based on the business status of each registered application when it detects that there is no registered application running in the foreground or the focus is not on the registered application.

[0154] Generally speaking, many electronic devices are equipped with voice assistant applications. When giving voice commands to the voice assistant application, usually no operations are performed on other applications. For example, when a user inputs the voice "What time is it now" into the electronic device, the breath wake-up engine application detects the breath wake-up event, and when it finds that the registered application is not running in the foreground or the focus is not on the registered application, it sends a breath wake-up event to the voice assistant application. After receiving the breath wake-up event, the voice assistant application obtains the voice "What time is it now" by fetching the stream, processes the voice data, finds out that the time is 10:49, and returns the time to the user.

[0155] By applying the method of the embodiment of the present application, when the registered application running in the foreground or the registered application in focus is not identified, the voice assistant application can be determined as the target application, thereby avoiding the situation where there is no application response to voice wake-up.

[0156] When the registered apps include input method apps and voice assistant apps;

[0157] Then step S502 may include:

[0158] The breath wake-up engine application determines the input method application as the target application based on the business status of each registered application, when it detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application; otherwise, it determines the voice assistant application as the target application.

[0159] When the input method application is running, a soft keyboard will be pulled up and the focus will be on the input box, such as Figure 8 As shown. When the breath wake-up engine application detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application, it is considered that the user is using voice input when inputting text, so the input method application is determined to be the target application; if the input method is not running, it is considered that the user is trying to wake up the voice assistant application, so the voice assistant is determined to be the target application.

[0160] In actual applications, when an input method application needs to perform voice input, the voice-to-text button needs to be pressed throughout the voice input process to convert the acquired voice into text for input, such as Figure 8 As shown, Figure 8 The box shows the voice-to-text button. In addition, the voice-to-text buttons of most input method applications are at the bottom of the screen. When users use one hand to operate, the bottom buttons are inconvenient to operate. Fig. 9As shown, the area marked "EASY&ACCURATE" is an area that is relatively easy to reach with one hand, the area marked "OKAY" is an area that can be reached with one hand, and the darker area is an area that cannot be reached with one hand or is more difficult to reach. It should be noted that the input method application in this application is not limited to the input method application called in the chat box or dialog box. Any input method application that can be called can be applied to this application, for example, it can be an input method application called when editing a search box or text box.

[0161] In one example, a user can Fig.10 Perform voice input as shown in the following operation: Fig.10 As shown, when inputting, the user manually calls up the input method soft keyboard and cursor. After pulling up the input method interface, the user can trigger a breath wake-up event to perform breath wake-up input, that is, voice input. For example, lift the phone to 3 cm from the mouth to perform voice input. After the breath wake-up engine application in the mobile phone recognizes the breath wake-up event, it queries the input method application for the soft keyboard to be pulled up, and the focus is on the input box of the input method application. Then, the voice data processing task is issued to the input method application. After receiving the voice data processing task, for example, if the voice processing task is input method voice-to-text, the input voice data is obtained, converted into text, and sent out according to the user's instructions.

[0162] By using the method of the embodiment of the present application, when the soft keyboard of the input method application is identified to be pulled up and the focus is on the input box of the input method application, the input method application can be determined as the target application, thereby waking up the input method application. The user can directly perform voice input, and the input method application can directly input according to the acquired voice data without the user having to press the voice-to-text button to input throughout the process, thereby improving the convenience of the user when inputting.

[0163] In a second aspect of an embodiment of the present application, a voice data processing device is provided, which is applied to an electronic device, wherein a breath wake-up engine application is running in the electronic device, and the device includes: Fig.11 The structure shown:

[0164] The service status query module 1101 is used for, when the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application;

[0165] The target application determination module 1102 is used for the breath awakening engine application to determine the target application among the registered applications according to the business status of the registered applications;

[0166] The task distribution module 1103 is used for the breath awakening engine application to distribute the voice data processing task to the target application;

[0167] The voice data processing module 1104 is used for the target application to obtain and process voice data according to the processing task.

[0168] The device using the embodiment of the present application can determine the target application from the registered applications by querying the business status of the registered applications when a breath wake-up event is detected, thereby waking up different application programs.

[0169] In a possible implementation manner, the voice data processing device in the embodiment of the present application may further include:

[0170] The breath awakening channel management module is used to open the breath awakening channel to detect the breath awakening event when the breath awakening engine application detects the access of at least one registered application; and close the breath awakening channel when the breath awakening engine application does not detect the access of the registered application.

[0171] The device using the embodiment of the present application can detect the access status of registered applications in advance to determine whether to open the breath wake-up path, and then close the breath wake-up path when no registered application is accessed, thereby reducing the power consumption of the electronic device.

[0172] In a possible implementation manner, the voice data processing device in the embodiment of the present application may further include:

[0173] A registration information sending module, used for, after the application to be registered is started, for the application to be registered to run the breath wake-up switch and maintain the state of its own breath wake-up switch, and to send a switch registration message to the breath wake-up engine application;

[0174] The application registration module is used for the breath wake-up engine to register the breath wake-up switch of the application to be registered according to the switch registration message; wherein, the application to be registered after the breath wake-up switch is successfully registered is a registered application.

[0175] The device using the embodiment of the present application can judge the state of the breath wake-up switch to determine whether the application is registered for the switch. After successful registration, the switch state can be used to judge the access status of the registered application, thereby determining whether to open the breath wake-up path. Further, after detecting the breath wake-up event, the voice data processing task can be sent to the registered application that is connected to the breath wake-up engine application.

[0176] In a possible implementation, the target application determination module may include:

[0177] The target application determination submodule is specifically used for the breath awakening engine application to select the registered application with the focus in the registered applications running in the foreground to obtain the target application according to the business status of each registered application.

[0178] The device using the embodiment of the present application can directly determine the registered application with focus in the foreground as the target application, thereby completing the distribution of voice processing tasks to the target application and the processing of voice data by the target application.

[0179] In a possible implementation manner, the voice data processing device in the embodiment of the present application may further include:

[0180] The registered application status query module is used for the breath wake-up engine application to determine the voice assistant application as the target application based on the business status of each registered application when it detects that there is no registered application running in the foreground or the focus is not on the registered application.

[0181] The device using the embodiment of the present application can determine the voice assistant application as the target application when the registered application running in the foreground or the registered application in focus is not identified, thereby avoiding the situation where there is no application response to voice wake-up.

[0182] In a possible implementation, the registered applications include an input method application and a voice assistant application;

[0183] The target application determination module may include:

[0184] The input method application status query module determines that the breath wake-up engine application is the target application based on the business status of each registered application. When it detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application, it determines that the input method application is the target application; otherwise, it determines that the voice assistant application is the target application.

[0185] The device using the embodiment of the present application can directly determine the input method application as the target application when recognizing that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application, thereby waking up the input method application. The user can directly perform voice input, and the input method application can directly input according to the acquired voice data without the user having to press the voice-to-text button to input throughout the process, thereby improving the convenience of the user when inputting.

[0186] In another embodiment provided in the present application, a computer-readable storage medium is provided, in which a computer program is stored. When the computer program is executed by a processor, the steps of any of the above-mentioned voice data processing methods are implemented.

[0187] In another embodiment provided by the present application, a computer program product including instructions is also provided, which, when executed on a computer, enables the computer to execute any of the speech data processing methods in the above embodiments.

[0188] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function described in the embodiment of the present application is generated in whole or in part. The computer may be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions may be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from a website site, computer, server or data center by wired (e.g., coaxial cable, optical fiber, digital subscriber line (DSL)) or wireless (e.g., infrared, wireless, microwave, etc.) mode to another website site, computer, server or data center. The computer-readable storage medium may be any available medium that a computer can access or a data storage device such as a server or data center that includes one or more available media integrated. The available medium may be a magnetic medium, (e.g., a floppy disk, a hard disk, a tape), an optical medium (e.g., a DVD), or a semiconductor medium (e.g., a solid-state drive Solid State Disk (SSD)), etc.

[0189] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0190] Each embodiment in this specification is described in a related manner, and the same or similar parts between the embodiments can be referred to each other, and each embodiment focuses on the differences from other embodiments. In particular, for the device and electronic device embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the partial description of the method embodiments.

[0191] The above description is only a preferred embodiment of the present application and is not intended to limit the protection scope of the present application. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application are included in the protection scope of the present application.

Claims

1. A method for processing speech data, characterized in that: Applied to an electronic device, wherein a breath awakening engine application is running in the electronic device, the method comprises: When the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application; The breath awakening engine application determines a target application from among the registered applications according to the service status of each of the registered applications; The breath awakening engine application distributes the processing task of the voice data to the target application; The target application acquires and processes the voice data according to the processing task.

2. The method according to claim 1, characterized in that The method further comprises: When the breath wake-up engine application detects access to at least one registered application, the breath wake-up path is opened to detect a breath wake-up event; when the breath wake-up engine application does not detect access to a registered application, the breath wake-up path is closed.

3. The method according to claim 2, characterized in that The method further comprises: After the application to be registered is started, the application to be registered runs the breath wake-up switch and maintains the state of its own breath wake-up switch, and sends a switch registration message to the breath wake-up engine application; The breath wake-up engine registers the breath wake-up switch of the application to be registered according to the switch registration message; wherein the application to be registered after the breath wake-up switch is successfully registered is the registered application.

4. The method according to claim 1, characterized in that: The breath awakening engine application determines a target application from among the registered applications according to the service status of each of the registered applications, including: The breath awakening engine application selects the registered application with focus from the registered applications running in the foreground to obtain the target application according to the business status of each of the registered applications.

5. The method according to claim 4, characterized in that The method further comprises: The breath awakening engine application determines the voice assistant application as the target application based on the business status of each registered application when it is detected that there is no registered application running in the foreground or the focus is not on the registered application.

6. The method according to claim 1, characterized in that The registered applications include input method applications and voice assistant applications; The breath awakening engine application determines a target application from among the registered applications according to the service status of each of the registered applications, including: The breath wake-up engine application determines that the input method application is the target application based on the business status of each registered application, when it detects that the soft keyboard of the input method application is pulled up and the focus is on the input box of the input method application; otherwise, it determines that the voice assistant application is the target application.

7. A voice data processing device, characterized in that: Applied to an electronic device, wherein a breath awakening engine application is running in the electronic device, the device comprises: A service status query module, used for, when the breath awakening engine application detects a breath awakening event, the breath awakening engine application obtains input voice data and queries the service status of each registered application that has completed registration with the breath awakening engine application; A target application determination module, used for the breath awakening engine application to determine a target application among the registered applications according to the business status of each registered application; A task distribution module, used for the breath awakening engine application to distribute the processing task of the voice data to the target application; The voice data processing module is used for the target application to obtain and process the voice data according to the processing task.

8. The device according to claim 7, characterized in that The device also includes: The breath awakening path management module is used to open the breath awakening path to detect the breath awakening event when the breath awakening engine application detects the access of at least one registered application; and close the breath awakening path when the breath awakening engine application does not detect the access of the registered application.

9. An electronic device, characterized in that: include: Memory, used to store computer programs; A processor, for implementing any of the methods described in claims 1-6 when executing a program stored in a memory.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 6 is implemented.

Citation Information

Patent Citations

  • Speech interactive awakening electronic device and method based on microphone signal and medium

    CN110223711A

  • Voice interaction processing method and device and electronic equipment

    CN111354360A

  • Voice wake-up method and electronic equipment

    CN113990323A

  • Voice control method, device and equipment and computer storage medium

    CN114639384A

  • Voice input method and device, electronic equipment and storage medium

    CN119028333A

Cited By

  • Voice input method and device, electronic equipment and storage medium

    CN119028333A

  • A voice input method, apparatus, electronic device, and storage medium

    CN119028333B