A voice control method, device and system

Abstract

The application provides a voice control method, device and system. When making a wake-up decision, the same wake-up word or different wake-up words use different wake-up thresholds in different use scenarios, so that the wake-up threshold changes with the use scenario, meeting the wake-up needs of users in different use scenarios and improving user experience. Users can configure the same or different wake-up words for different electronic devices through a management device, or customize a wake-up word through the management device and configure different use scenarios (such as use modes or device modes) for the customized wake-up word, further meeting the wake-up needs of users in different use scenarios.

Description

Technical Field

[0001] This application relates to the field of voice control technology, and in particular to a voice control method, device and system. Background Technology

[0002] With the rapid development of mobile devices, voice interaction has become increasingly popular. Most electronic devices now offer voice assistants for voice control. However, before voice interaction can occur, a wake word is often needed to activate the device, followed by voice recognition. Therefore, as the entry point for human-computer interaction, the ease of use and accuracy of voice wake-up are crucial.

[0003] Currently, electronic devices generally use built-in wake words to wake up, which to some extent cannot meet users' needs for waking up electronic devices. Summary of the Invention

[0004] This application provides a voice control method, apparatus, and system to meet users' needs for waking up electronic devices.

[0005] Firstly, this application provides a voice control method, which may be implemented by a first electronic device or by a chip or chip system of the first electronic device. The method includes: monitoring voice information; when determining that the probability of the voice information containing a first wake-up word is greater than a first wake-up threshold, performing a first operation to wake up the first electronic device corresponding to the first wake-up word in a first usage scenario; the first wake-up word is a wake-up word configured in the first electronic device in the first usage scenario, and the first wake-up threshold is the wake-up threshold of the first wake-up word in the first usage scenario; the wake-up threshold is different in different usage scenarios. When determining that the probability of the voice information containing a second wake-up word is greater than a second wake-up threshold, performing a second operation to wake up the first electronic device corresponding to the second wake-up word in a second usage scenario; the second wake-up word is a wake-up word configured in the first electronic device in the second usage scenario, and the second wake-up threshold is the wake-up threshold of the second wake-up word in the second usage scenario; wherein the first wake-up threshold is different from the second wake-up threshold. Through the above scheme, the wake-up threshold is different in different usage scenarios, thus the wake-up threshold changes with the usage scenario, which can meet the user's wake-up needs in different usage scenarios and improve the user experience.

[0006] In one possible design, the first wake word is the same as the second wake word, or the first wake word is different from the second wake word.

[0007] For example, when the first wake word and the second wake word are different, the usage scenarios corresponding to the different wake words can be different. Thus, the first electronic device can determine the usage scenario through the wake word.

[0008] In one possible design, when the first electronic device is connected to the first local area network, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is connected to the second local area network, the usage scenario of the first electronic device is the second usage scenario; or, when the first electronic device is located in the first location area, the usage scenario of the first electronic device is the first usage scenario, and when the first electronic device is located in the second location area, the usage scenario of the first electronic device is the second usage scenario.

[0009] In the above design, for the same or different wake words, different usage scenarios correspond to different local area networks (LANs) or different location areas. This allows electronic devices to determine the current usage scenario based on the currently accessed LAN or location area. Consequently, different location areas or different LANs utilize different usage scenarios, further enabling automatic switching of the wake-up threshold based on user needs and improving the user experience.

[0010] In one possible design, before determining that the probability of the voice information containing the first wake-up word is greater than a first wake-up threshold, the method further includes: acquiring information input by the user, the information including wake-up word data of the first wake-up word and usage scenario information of the first wake-up word, wherein the usage scenario information indicates the first usage scenario; the wake-up word data and the usage scenario information are used to determine the first wake-up threshold.

[0011] The above design allows users to configure usage scenario information according to their needs, thereby improving the user experience.

[0012] In one possible design, the information input by the user may also include one or more of the following: location area information, time period information, or wireless LAN information.

[0013] In one possible design, the electronic device can associate and store the location area information, time period information, or wireless LAN information of the first wake word with the first wake word, the first wake-up threshold, and the usage scenario information.

[0014] In one possible design, the method further includes: sending user input information, including the wake word data and the usage scenario information, to a server; and receiving a first wake-up threshold determined by the server based on the wake word data and the usage scenario information.

[0015] In the above design, the wake-up threshold is determined by a server, which reduces the resource consumption of electronic devices.

[0016] In one possible design, the method further includes: performing synthetic processing on the wake-up word data according to the usage scenario information to obtain a synthetic speech dataset; and determining a first wake-up threshold for the first wake-up word in a first usage scenario based on the synthetic speech dataset.

[0017] In the above design, scene information is used to synthesize data for determining the wake-up threshold, thereby making the wake-up threshold change with the usage scenario, which is highly flexible.

[0018] In one possible design, the usage scenario information includes usage mode and / or device mode; wherein, the usage mode indicates a private mode or a public mode, the private mode is used to indicate that the first wake word is adapted to a private user, and the public mode is used to indicate that the first wake word is adapted to multiple users; the device mode indicates a near-field mode or a far-field mode, the wake-up distance of the near-field mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

[0019] In one possible design, the usage scenario information includes the usage mode, which indicates a public mode. The wake-up word data is synthesized based on the usage scenario information to obtain a synthesized speech dataset, including: if the wake-up word data is in text format, performing text-to-speech (TTS) processing on the wake-up word data to obtain first speech data, and performing data augmentation processing on the first speech data to obtain multiple first speech synthesis data sets, the synthesized dataset including the multiple first speech synthesis data sets and the first speech data; or, if the wake-up word data is in speech format, performing data augmentation processing on the wake-up word data to obtain multiple second speech synthesis data sets; the synthesized dataset including the multiple second speech synthesis data sets and the wake-up word data.

[0020] In the above design, data diversity is ensured by augmenting the data, such as changing the tone, emotion, and intonation of the generated data to generate data with different noise, reverberation, speech rate, and volume.

[0021] In one possible design, the usage scenario information includes the usage mode, which indicates a private mode, and the wake-up word data is in speech format. The wake-up word data is synthesized based on the usage scenario information to obtain a synthesized speech dataset, including: performing voice cloning processing on the wake-up word data to obtain multiple third-party speech synthesis data; the synthesized dataset includes the multiple third-party speech synthesis data and the wake-up word data. In the above design, voice cloning technology is used to generate data that mimics the user's voice, adding effective data to determine the wake-up threshold and improving the accuracy of the determined wake-up threshold.

[0022] In one possible design, the usage scenario information includes a usage mode and a device mode, wherein the usage mode indicates a public mode and the device mode indicates a far-field speaking mode; the wake word data is synthesized according to the usage scenario information to obtain a synthesized speech dataset, including: if the wake word data is in text format, performing TTS processing on the wake word data to obtain second speech data, performing data augmentation processing on the second speech data to obtain multiple fourth speech synthesis data, and converting the fourth speech synthesis data and the second speech data into far-field speaking speech to obtain the synthesized speech dataset; or, if the wake word data is in speech format, performing data augmentation processing on the wake word data to obtain multiple fifth speech synthesis data, and converting the fourth speech synthesis data and the wake word data into far-field speaking speech to obtain the synthesized speech dataset.

[0023] In one possible design, obtaining user input information includes: displaying a first user interface, the first user interface including an input control for the user to input a first wake-up word and a first selection control for selecting a usage scenario for the first wake-up word; in response to the user's operation of inputting the first wake-up word through the input control and selecting a first usage scenario through the first selection control, obtaining the wake-up word data and the usage scenario information of the first wake-up word.

[0024] In one possible design, the method further includes: displaying a main control interface, the main control interface including a first option for adding a custom wake word; and displaying the first user interface in response to a user selecting the first option.

[0025] In one possible design, the main control interface further includes a second option for selecting N wake words defined for the first electronic device; in response to the operation of selecting the second option, a third user interface is displayed; wherein the third user interface includes N wake word options, and different wake word options are used to select different wake words; the third user interface further includes a second selection control for selecting the usage scenario corresponding to each wake word.

[0026] The above design supports the selection of wake words, improving the user experience.

[0027] In one possible design, determining that the probability of the voice information containing the first wake-up word is greater than a first wake-up threshold includes: determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold based on a probability compensation value set; wherein, the probability compensation value set includes probability compensation values ​​of N1 modeling units used to determine whether the voice information contains the first wake-up word in the first usage scenario, where N1 is a positive integer; the probability compensation value of the first modeling unit is used to compensate the acoustic posterior probability of the first modeling unit when determining the acoustic posterior probability of the voice information containing the first modeling unit, the acoustic posterior probability of the first modeling unit is used to determine the probability of the first wake-up word, and the first modeling unit is any one of the N1 modeling units.

[0028] The above design corrects the wake-up model by performing probability compensation on the modeling unit, which is simple and effective.

[0029] In one possible design, the method further includes: determining a set of probability compensation values ​​for the first wake-up word in a first usage scenario based on the synthesized speech dataset; wherein the synthesized speech dataset is obtained by synthesizing wake-up word data of the first wake-up word input by the user based on usage scenario information used to indicate the first usage scenario.

[0030] In one possible design, the method further includes: receiving a set of probability compensation values ​​for the first wake-up word in a first usage scenario sent by the server.

[0031] In one possible design, determining the probability compensation value set of the first wake-up word in a first usage scenario based on a synthesized speech dataset, wherein the synthesized speech dataset includes K synthesized speech information, includes: extracting speech feature information of each basic computing unit in the K synthesized speech data; determining the acoustic posterior probability of the modeling unit corresponding to M basic computing units based on the speech feature data of each basic computing unit; the modeling unit corresponding to the M basic computing units is the modeling unit included in the first wake-up word; obtaining the average value of the acoustic posterior probability of the j-th modeling unit corresponding to the M basic computing units; the i-th modeling unit is the j-th of the N1 modeling units, where j is a positive integer less than or equal to N1; obtaining the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units; and obtaining the probability compensation value of the j-th modeling unit based on the average value of the acoustic posterior probability of the j-th modeling unit and the maximum value.

[0032] In one possible design, determining the first wake-up threshold for the first wake-up word in a first usage scenario based on the synthesized speech dataset includes: performing probability compensation on the acoustic posterior probabilities of the M basic computing units corresponding to the j-th modeling unit based on the probability compensation value of the j-th modeling unit to obtain the compensated acoustic posterior probability of the j-th modeling unit, where j takes all positive integers less than or equal to N1; determining the probability that the synthesized speech dataset includes the first wake-up word based on the compensated acoustic posterior probabilities of the N1 modeling units, wherein the probability that the synthesized speech dataset includes the first wake-up word is used to determine the first wake-up threshold.

[0033] In one possible design, before determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold, the method further includes: receiving a control message sent by a server, the control message including the first wake-up word, usage scenario information for indicating a first usage scenario of the first wake-up word, and a first wake-up threshold of the first wake-up word in the first usage scenario.

[0034] In one possible design, determining that the probability of the voice information containing a first wake-up word is greater than a first wake-up threshold includes: receiving the voice information detected by a microphone through a frame layer object; determining, through the frame layer object, at least one wake-up word configured on the first electronic device and usage scenario information corresponding to the at least one wake-up word, wherein the at least one wake-up word includes the first wake-up word; obtaining, through the frame layer object, a wake-up threshold corresponding to the usage scenario information of each stored wake-up word; and determining, through the frame layer object, that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold and the probability of the voice information containing other wake-up words is less than or equal to the wake-up threshold of the other wake-up words.

[0035] Secondly, embodiments of this application also provide a voice control device applied in an electronic device. For example, the voice control device can be an electronic device or a chip or chip system applied to an electronic device. The voice control device includes corresponding functional modules, each used to implement the steps in the above methods, as detailed in the method examples, which will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions. For example, the voice control device includes a listening module and a processing module. A monitoring module is used to monitor voice information; a processing module is used to determine that when the probability of the voice information containing a first wake-up word is greater than a first wake-up threshold, perform a first operation to wake up the first electronic device corresponding to the first wake-up word in a first usage scenario; the first wake-up word is a wake-up word configured in the first electronic device in the first usage scenario, and the first wake-up threshold is the wake-up threshold of the first wake-up word in the first usage scenario; or, the processing module is used to determine that when the probability of the voice information containing a second wake-up word is greater than a second wake-up threshold, perform a second operation to wake up the first electronic device corresponding to the second wake-up word in a second usage scenario; the second wake-up word is a wake-up word configured in the first electronic device in the second usage scenario, and the second wake-up threshold is the wake-up threshold of the second wake-up word in the second usage scenario; wherein, the first wake-up threshold is different from the second wake-up threshold.

[0036] Thirdly, embodiments of this application provide a voice control device, including a memory and a processor; the memory is used to store programs or instructions; the processor is used to invoke the programs or instructions to cause the device to execute the method described in the first aspect or any design of the first aspect.

[0037] Fourthly, an embodiment of this application provides a voice control method, comprising: configuring a first wake-up word and a first wake-up threshold corresponding to the first wake-up word to a first electronic device, wherein the first wake-up threshold is a wake-up threshold of the first wake-up word in a first usage scenario; configuring a second wake-up word and a second wake-up threshold corresponding to the second wake-up word to the first electronic device; wherein the second wake-up threshold is a wake-up threshold of the second wake-up word in a second usage scenario; wherein the first wake-up threshold is different from the second wake-up threshold.

[0038] In one possible design, the first wake word is different from the second wake word.

[0039] In one possible design, the method further includes: configuring a third wake-up word and a third wake-up threshold corresponding to the third wake-up word to the third electronic device, wherein the third wake-up threshold is the wake-up threshold of the third wake-up word in a third usage scenario.

[0040] In the above design, different electronic devices are configured with different wake words and wake thresholds to reduce false wake-ups.

[0041] In one possible design, the method further includes: acquiring user input information, the information including wake word data of the first wake word and usage scenario information of the first wake word, wherein the usage scenario information indicates the first usage scenario; performing synthesis processing on the wake word data according to the usage scenario information to obtain a synthesized speech dataset; and determining a first wake-up threshold of the first wake word in the first usage scenario according to the synthesized speech dataset.

[0042] In one possible design, the method further includes: determining a set of probability compensation values ​​for the first wake-up word in a first usage scenario based on a synthesized speech dataset; wherein the synthesized speech dataset is obtained by synthesizing wake-up word data of the first wake-up word input by the user based on usage scenario information input by the user to indicate the first usage scenario. The set of probability compensation values ​​is then sent to the first electronic device.

[0043] In one possible design, determining the probability compensation value set of the first wake-up word in a first usage scenario based on the synthesized speech dataset includes: extracting speech feature information of each basic computational unit in the K synthesized speech data; determining the acoustic posterior probability of the modeling unit corresponding to the M basic computational units based on the speech feature data of each basic computational unit; the modeling unit corresponding to the M basic computational units is the modeling unit included in the first wake-up word; obtaining the average value of the acoustic posterior probability of the j-th modeling unit corresponding to the M basic computational units; the j-th modeling unit is the j-th of the N1 modeling units, where j is a positive integer less than or equal to N1; obtaining the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units; and obtaining the probability compensation value of the j-th modeling unit based on the average value of the acoustic posterior probability of the j-th modeling unit and the maximum value.

[0044] In one possible design, determining the first wake-up threshold for the first wake-up word in a first usage scenario based on the synthesized speech dataset includes: performing probability compensation on the acoustic posterior probabilities of the M basic computing units corresponding to the j-th modeling unit based on the probability compensation value of the j-th modeling unit to obtain the compensated acoustic posterior probability of the j-th modeling unit, where j takes all positive integers less than or equal to N1; determining the probability that the synthesized speech dataset includes the first wake-up word based on the compensated acoustic posterior probabilities of the N1 modeling units, where the probability that the synthesized speech dataset includes the first wake-up word is used to determine the first wake-up threshold.

[0045] Fifthly, embodiments of this application also provide a voice control device that can be applied to management devices. For example, the voice control device can be a management device or a chip or chip system applied to a management device. The management device can be an electronic device or a server. The voice control device includes corresponding functional modules, each used to implement the steps in the method of the fourth aspect or any of the designs in the fourth aspect above. For details, please refer to the detailed description in the method examples, which will not be repeated here. The functions can be implemented by hardware or by hardware executing corresponding software. The hardware or software includes one or more modules corresponding to the above functions.

[0046] In a sixth aspect, embodiments of this application provide a voice control device, including a memory and a processor; the memory is used to store programs or instructions; the processor is used to invoke the programs or instructions to cause the device to execute the method described in the fourth aspect or any design of the fourth aspect.

[0047] In a seventh aspect, this application provides a computer-readable storage medium storing a computer program or instructions that, when executed by an electronic device, cause the electronic device to perform the methods described in the first aspect or any possible design of the first aspect, or cause the electronic device to perform the methods described in the fourth aspect or any possible design of the fourth aspect.

[0048] Eighthly, this application provides a computer program product comprising a computer program or instructions that, when executed by an electronic device, implement the method in the first aspect or any possible implementation of the first aspect, or implement the method in the fourth aspect or any possible design of the fourth aspect.

[0049] Ninthly, embodiments of this application provide a voice control system, including a management device and a first electronic device, wherein the management device establishes a communication connection with the first electronic device; the management device is configured to configure a first wake-up word and a first wake-up threshold corresponding to the first wake-up word to the first electronic device, wherein the first wake-up threshold is a wake-up threshold of the first wake-up word in a first usage scenario; the first electronic device is configured to perform a first operation to wake up the first electronic device corresponding to the first wake-up word in the first usage scenario when the probability of detecting that the first voice information includes the first wake-up word is greater than the first wake-up threshold; the management device is further configured to configure a second wake-up word and a second wake-up threshold corresponding to the second wake-up word to the first electronic device; the second wake-up threshold is a wake-up threshold of the second wake-up word in a second usage scenario; the first electronic device is further configured to perform a second operation to wake up the second electronic device corresponding to the second wake-up word in the second usage scenario when the probability of detecting that the second voice information includes the second wake-up word is greater than the second wake-up threshold; wherein the first wake-up threshold is different from the second wake-up threshold.

[0050] In one possible design, the first wake word is different from the second wake word.

[0051] In one possible design, when the first electronic device is connected to the first local area network, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is connected to the second local area network, the usage scenario of the first electronic device is the second usage scenario; or, when the first electronic device is located in the first location area, the usage scenario of the first electronic device is the first usage scenario, and when the first electronic device is located in the second location area, the usage scenario of the first electronic device is the second usage scenario.

[0052] In one possible design, the management device is a second electronic device, which is further configured to: before determining that the probability of the first voice information containing the first wake-up word is greater than a first wake-up threshold, acquire information input by the user, the information including wake-up word data of the first wake-up word and usage scenario information of the first wake-up word, wherein the usage scenario information indicates the first usage scenario; determine the first wake-up threshold based on the wake-up word data and the usage scenario information, and send the first wake-up threshold to the first electronic device.

[0053] In the above design, different electronic devices are configured with the same wake word and wake-up threshold. This allows users to wake up multiple electronic devices simultaneously.

[0054] In one possible design, the management device is a server, and the system further includes a third electronic device; the management device is also configured to configure a third wake-up word and a third wake-up threshold corresponding to the third wake-up word to the third electronic device, wherein the third wake-up threshold is a wake-up threshold for the third wake-up word in a third usage scenario; the third electronic device is also configured to perform a third operation to wake up the third electronic device corresponding to the third wake-up word in the third usage scenario when the probability of detecting third voice information including the third wake-up word is greater than the third wake-up threshold; wherein the third wake-up threshold is different from the first wake-up threshold and the second wake-up threshold.

[0055] In the above design, different electronic devices are configured with different wake words and wake thresholds to reduce false wake-ups.

[0056] In one possible design, the management device is a server, and the system further includes a fourth electronic device; the fourth electronic device is used to acquire information input by the user, the information including wake word data of the first wake word and usage scenario information of the first wake word, wherein the usage scenario information indicates the first usage scenario; and is used to send the user input information to the server; the server is used to determine the first wake-up threshold based on the wake word data and the usage scenario information, and send the first wake-up threshold to the first electronic device.

[0057] In one possible design, the information input by the user may also include information about the electronic device to which the first wake word is applied, and the electronic device information indicates the first electronic device.

[0058] In one possible design, the system further includes a fifth electronic device; the electronic device information also indicates the fifth electronic device; the server is further configured to send the first wake word and the first wake threshold to the fifth electronic device.

[0059] In one possible design, the server is specifically configured to: perform synthetic processing on the wake-up word data according to the usage scenario information to obtain a synthetic speech dataset; and determine a first wake-up threshold for the first wake-up word in a first usage scenario based on the synthetic speech dataset.

[0060] In one possible design, the usage scenario information includes usage mode and / or device mode; wherein, the usage mode indicates a private mode or a public mode, the private mode is used to indicate that the first wake word is adapted to a private user, and the public mode is used to indicate that the first wake word is adapted to multiple users; the device mode indicates a near-field mode or a far-field mode, the wake-up distance of the near-field mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

[0061] In one possible design, the usage scenario information includes the usage mode, which indicates a public mode. The server is specifically configured to: if the wake-up word data is in text format, perform text-to-speech (TTS) processing on the wake-up word data to obtain first speech data, and perform data augmentation processing on the first speech data to obtain multiple first speech synthesis data, wherein the synthesis dataset includes the multiple first speech synthesis data and the first speech data; or, if the wake-up word data is in speech format, perform data augmentation processing on the wake-up word data to obtain multiple second speech synthesis data, wherein the synthesis dataset includes the multiple second speech synthesis data and the wake-up word data.

[0062] In one possible design, the usage scenario information includes the usage mode, the usage mode indicating a private mode, and the wake word data is in speech format; the server is specifically used to: perform sound cloning processing on the wake word data to obtain multiple third-party speech synthesis data; the synthesis dataset includes the multiple third-party speech synthesis data and the wake word data.

[0063] In one possible design, the usage scenario information includes a usage mode and a device mode, wherein the usage mode indicates a public mode and the device mode indicates a far-field speaking mode; the server is specifically configured to: when the wake-up word data is in text format, perform TTS processing on the wake-up word data to obtain second speech data, perform data augmentation processing on the second speech data to obtain multiple fourth speech synthesis data, and convert the fourth speech synthesis data and the second speech data into far-field speaking speech to obtain the synthesized speech dataset; or, when the wake-up word data is in speech format, perform data augmentation processing on the wake-up word data to obtain multiple fifth speech synthesis data, and convert the fourth speech synthesis data and the wake-up word data into far-field speaking speech to obtain the synthesized speech dataset.

[0064] In one possible design, a management device is specifically configured to determine a set of probability compensation values ​​for the first wake-up word in a first usage scenario based on a synthesized speech dataset; wherein the synthesized speech dataset is obtained by synthesizing wake-up word data of the first wake-up word input by the user based on usage scenario information input by the user to indicate the first usage scenario.

[0065] A first electronic device is specifically configured to determine, based on a set of probability compensation values, that the probability of the voice information including a first wake-up word is greater than a first wake-up threshold. The set of probability compensation values ​​includes probability compensation values ​​for N1 modeling units used in the first usage scenario to determine whether the voice information includes the first wake-up word, where N1 is a positive integer. The probability compensation value of the first modeling unit is used to compensate for the acoustic posterior probability of the first modeling unit when determining that the voice information includes the first modeling unit. The acoustic posterior probability of the first modeling unit is used to determine the probability of the first wake-up word, and the first modeling unit is any one of the N1 modeling units. This design, by performing probability compensation on the modeling units, corrects the wake-up model in a simple and effective way.

[0066] In one possible design, the management device is specifically used to extract speech feature information of each basic computing unit in the K synthesized speech data; determine the acoustic posterior probability of the modeling unit corresponding to the M basic computing units based on the speech feature data of each basic computing unit; the modeling unit corresponding to the M basic computing units is the modeling unit included in the first wake-up word; obtain the average value of the acoustic posterior probability of the j-th modeling unit corresponding to the M basic computing units; the j-th modeling unit is the j-th of the N1 modeling units, where j is a positive integer less than or equal to N1; obtain the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units; and obtain the probability compensation value of the j-th modeling unit based on the average value of the acoustic posterior probability of the j-th modeling unit and the maximum value.

[0067] In one possible design, the management device performs probability compensation on the acoustic posterior probabilities of the M basic computing units corresponding to the j-th modeling unit based on the probability compensation value of the j-th modeling unit to obtain the compensated acoustic posterior probability of the j-th modeling unit, where j takes all positive integers less than or equal to N1; the probability that the synthesized speech dataset includes the first wake-up word is determined based on the compensated acoustic posterior probabilities of the N1 modeling units, and the probability that the synthesized speech dataset includes the first wake-up word is used to determine the first wake-up threshold.

[0068] In one possible design, the first electronic device, when determining that the probability of the voice information containing a first wake-up word is greater than a first wake-up threshold, is specifically configured to receive the voice information detected by the microphone through a frame layer object; determine, through the frame layer object, at least one wake-up word configured in the first electronic device and usage scenario information corresponding to the at least one wake-up word, wherein the at least one wake-up word includes the first wake-up word; obtain, through the frame layer object, a wake-up threshold corresponding to the usage scenario information of each stored wake-up word; and determine, through the frame layer object, that the probability of the voice information including the first wake-up word is greater than the first wake-up threshold and the probability of the voice information including other wake-up words is less than or equal to the wake-up threshold of the other wake-up words.

[0069] In one possible design, the fourth electronic device is specifically configured to: display a first user interface, the first user interface including an input control for a user to input a first wake-up word and a first selection control for selecting a usage scenario for the first wake-up word; and in response to the user's operation of inputting the first wake-up word through the input control and selecting a first usage scenario through the first selection control, acquire the wake-up word data and the usage scenario information of the first wake-up word.

[0070] In one possible design, the management device is a server, and the fourth electronic device is further configured to: display a second user interface, the second user interface including a first option for adding a custom wake word; and display the first user interface in response to a user selecting the first option.

[0071] In one possible design, the management device is a server, and the system further includes a fourth electronic device; the fourth electronic device is used to display a third user interface, the third user interface including a second option for selecting N wake words defined by the first electronic device for management; in response to the operation of selecting the second option, the fourth user interface is displayed; wherein, the fourth user interface includes N wake word options, different wake word options are used to select different wake words, and the N wake word options include options of the first wake word; the fourth electronic device is also used to send control information to the management device in response to selecting the second wake word option, the control information being used to instruct the configuration of a second wake word for the first electronic device; the management device is specifically used to configure a second wake word and a second wake-up threshold corresponding to the second wake word for the first electronic device according to the control information.

[0072] In this embodiment of the application, the wake word of the electronic device can be customized through the terminal device, and a private mode or a public mode can be configured for the wake word. For example, if the electronic device is a mobile phone, a private mode can be used, and if the electronic device is a home device, a public mode can be used, thus supporting the modification of the wake word of the electronic device.

[0073] The technical effects that can be achieved by any of the second to ninth aspects mentioned above can be referred to the description of the beneficial effects in the first aspect mentioned above, and will not be repeated here. Attached Figure Description

[0074] Figure 1 This is a schematic diagram of a possible voice control system architecture in an embodiment of this application;

[0075] Figure 2 This is a schematic diagram of a possible electronic device 100 in an embodiment of this application;

[0076] Figure 3 This is a schematic diagram of a possible terminal device 200 in an embodiment of this application;

[0077] Figure 4A This is a schematic diagram of the software structure of a possible terminal device 200 in an embodiment of this application;

[0078] Figure 4B This is a schematic diagram of the software structure of another possible terminal device 200 in the embodiments of this application;

[0079] Figure 5 This is a schematic diagram of the logical architecture of a possible voice control system in an embodiment of this application;

[0080] Figure 6 This is a schematic diagram of a first user interface in a first possible application scenario of the embodiments of this application;

[0081] Figure 7A This is a schematic diagram of a second user interface under a first possible application scenario of the embodiments of this application;

[0082] Figure 7B This is a schematic diagram of a third user interface under the first possible application scenario of the embodiments of this application;

[0083] Figure 7C This is a schematic diagram of the fourth user interface under the first possible application scenario of the embodiments of this application;

[0084] Figure 8A This is a schematic diagram of the fifth user interface under the first possible application scenario of the embodiments of this application;

[0085] Figure 8BThis is a schematic diagram of the sixth user interface under the first possible application scenario of the embodiments of this application;

[0086] Figure 8C This is a schematic diagram of the seventh user interface under the first possible application scenario of the embodiments of this application;

[0087] Figure 8D This is a schematic diagram of the eighth user interface under the first possible application scenario of the embodiments of this application;

[0088] Figure 8E This is a schematic diagram of the ninth user interface under the first possible application scenario of the embodiments of this application;

[0089] Figure 8F This is a schematic diagram of the tenth user interface under the first possible application scenario of the embodiments of this application;

[0090] Figure 9 This is a schematic diagram of the process for determining the wake-up threshold in an embodiment of this application;

[0091] Figure 10 This is a schematic diagram of the first user interface in a second possible application scenario of the embodiments of this application;

[0092] Figure 11 This is a schematic diagram of a second user interface in a second possible application scenario of the embodiments of this application;

[0093] Figure 12A This is a schematic diagram of a third user interface under a second possible application scenario of the embodiments of this application;

[0094] Figure 12B This is a schematic diagram of the fourth user interface under the second possible application scenario of the embodiments of this application;

[0095] Figure 12C This is a schematic diagram of the fifth user interface under the second possible application scenario of the embodiments of this application;

[0096] Figure 13 This is a schematic diagram of the sixth user interface under the second possible application scenario of the embodiments of this application;

[0097] Figure 14 This is a schematic diagram of the seventh user interface under the second possible application scenario of the embodiments of this application;

[0098] Figure 15 This is a schematic flowchart of the voice control method according to an embodiment of this application;

[0099] Figure 16 This is a schematic diagram of the voice control device 1600 according to an embodiment of this application;

[0100] Figure 17This is a schematic diagram of the voice control device 1700 according to an embodiment of this application. Detailed Implementation

[0101] The technical solutions in the embodiments of this application will now be described clearly and in detail with reference to the accompanying drawings.

[0102] The following explanations will first clarify some of the terms used in the embodiments of this application, so that those skilled in the art can understand them.

[0103] 1) The usage modes of a custom wake word can include private mode and public mode. Private mode, also known as private device mode or other names, is not specifically limited in this application. Private mode refers to an electronic device being used exclusively by one user. The custom wake word is only used by the private user, and other users cannot wake up the electronic device using this wake word. Public mode, also known as shared mode or other names, is not specifically limited in this application. In public mode, an electronic device can be used by multiple people, such as smart home devices like smart TVs, smart speakers, and smart curtains. In shared mode, a custom wake word can be used by multiple users to wake up the electronic device.

[0104] 2) Device modes can include proximity mode and distance mode. Proximity mode refers to the electronic device being used at close range, where it can be woken up by a wake word, such as within 30 centimeters. Distance mode refers to the electronic device being used at a distance, where it can be woken up by a wake word, such as 1 to 3 meters.

[0105] The near-field mode and the far-field mode are relative. For example, when using the far-field mode, the electronic device can still be woken up when the user is more than a certain distance away from it. However, when using the near-field mode, the electronic device may not be able to be woken up when the user is more than a certain distance away from it.

[0106] The near-field mode can also be called the near-field mode, and the far-field mode can also be called the far-field mode. Of course, it can also be named by other names. This application does not specifically limit this.

[0107] 3) Multi-device collaboration mode refers to a wake word that can be configured for multiple electronic devices. These devices can share the same wake word, and when the user speaks the wake word, multiple devices can be activated. Multi-device collaboration mode can also be called synchronization mode.

[0108] 4) The applications (apps) involved in the embodiments of this application are software programs capable of performing one or more specific functions. Typically, multiple applications can be installed on a terminal device. Examples include camera applications, email applications, and headphone control applications. The applications mentioned below can be system applications pre-installed on the terminal device at the factory, or third-party applications downloaded by the user from the network or obtained from other terminal devices during the use of the terminal device.

[0109] 5) The wake-up threshold involved in the embodiments of this application can also be called a confidence threshold, or other names. A wake-up threshold for a wake-up word is used to indicate the minimum probability that the determined voice data includes the wake-up word when deciding to wake up the electronic device. That is, the electronic device is determined to be woken up when the probability of detecting voice data including the wake-up word is greater than or equal to the wake-up threshold.

[0110] 6) In the embodiments of this application, "at least one (item)" refers to one (item) or more (items), and "multiple (items)" refers to two (items) or more than two (items). "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A existing alone, A and B existing simultaneously, or B existing alone, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects have an "or" relationship. "At least one of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one of a, b, or c can represent: a, b, c, ab, ac, bc, or abc, where a, b, and c can be single or multiple. In this application, the symbol "(a, b)" represents an open interval, with a range greater than a and less than b; "[a, b]" represents a closed interval, with a range greater than or equal to a and less than or equal to b; "(a, b]" represents a half-open interval, with a range greater than a and less than or equal to b; and "(a, b]" represents a half-open interval, with a range greater than a and less than or equal to b. Furthermore, unless otherwise stated, the ordinal numbers such as "first" and "second" mentioned in the embodiments of this application are used to distinguish multiple objects and are not used to limit the size, content, order, timing, priority, or importance of multiple objects.

[0111] The voice control method provided in this application can be applied to one or a group of electronic devices. A group of electronic devices may include multiple electronic devices, all of which have voice control functions. The wake-up words used by the multiple devices may be the same or different. Currently, devices with voice control functions usually have a fixed wake-up word, such as "Xiao Yi Xiao Yi". During use, the user wakes up the device by saying the wake-up word "Xiao Yi Xiao Yi".

[0112] In the first possible application scenario of this application, users can configure one or more custom wake words and their corresponding attributes (including usage scenario information) for a terminal device or electronic device according to their needs, such as actions corresponding to the custom wake word, such as device startup. For example, if the custom wake word is "Xiao Ke Xiao Ke", the user can wake up the terminal device or electronic device by saying the wake word "Xiao Ke Xiao Ke".

[0113] Usage scenario information indicates the usage scenario of the custom wake word. For example, the usage scenario includes user-private, multi-user shared, long-distance wake-up, short-distance wake-up, etc.

[0114] In this first possible application scenario, a terminal device can configure a custom wake word for itself or other electronic devices. It should be noted that configuring a custom wake word for an electronic device, as mentioned in this application, involves sending the custom wake word and its corresponding wake-up threshold to the electronic device.

[0115] In the second possible application scenario, users can manage wake words for multiple electronic devices in a unified manner, allowing several devices to share the same wake word and supporting the querying and updating of wake words. For example, a user can set a custom wake word "Xiao Bao Xiao Bao," which can be used to simultaneously wake up device 1, device 2, and device 3. This custom wake word can be applied to only one user or to multiple users. For instance, if it's applied to one user, when that user says the wake word "Xiao Bao Xiao Bao," it can simultaneously wake up device 1, device 2, and device 3. The actions performed by device 1, device 2, and device 3 when the wake word "Xiao Bao Xiao Bao" is used can be the same or different.

[0116] In this embodiment of the application, different wake words can be used to wake up the same or different electronic devices. In order to improve accuracy and reduce false wake-ups, a wake-up threshold is determined for each different custom wake word.

[0117] The method for determining the wake-up threshold will be explained in detail later, and will not be repeated here.

[0118] As an example, in the first possible application scenario, the terminal device can configure a custom wake-up word. The device that determines the wake-up threshold can be either the terminal device or a server. The electronic device affected by the custom wake-up word is also the terminal device. When the device determining the wake-up threshold is a server, the user can set a custom wake-up word and its usage scenario through the terminal device. The server then determines the wake-up threshold based on the custom wake-up word and the usage scenario and sends it to the terminal device. When the user speaks the custom wake-up word, the electronic device monitors the surrounding voice information and, based on the wake-up threshold indicated by the usage scenario information, makes a voice wake-up decision to determine whether the voice information includes the custom wake-up word.

[0119] As another example, in the first possible application scenario, the terminal device can configure a custom wake-up word for other electronic devices. The device determining the wake-up threshold can be either the terminal device or a server. When the device determining the wake-up threshold is a server, the user can set a custom wake-up word and its usage scenarios through the terminal device. The server then determines the wake-up threshold based on the custom wake-up word and usage scenarios before sending it to other electronic devices. Similarly, when the terminal device sets a custom wake-up word and its usage scenarios, the terminal device determines the wake-up threshold based on the custom wake-up word and usage scenarios before sending it to other electronic devices. Furthermore, when the user speaks the custom wake-up word, the electronic device can determine voice wake-up based on the wake-up threshold indicated by the usage scenario information.

[0120] As another example, in the second possible application scenario, the server obtains the user's input custom wake-up word, usage scenario, and multiple electronic devices affected by the custom wake-up word from the terminal device, confirms the wake-up threshold of the custom wake-up word, and sends the confirmed wake-up threshold of the custom wake-up word to the multiple electronic devices affected by the custom wake-up word. Then, when any user affected by the custom wake-up word speaks the custom wake-up word, any electronic device affected by the custom wake-up word can make a voice wake-up decision based on the confirmed wake-up threshold when it detects that the user has spoken the custom wake-up word.

[0121] The implementation methods for each of the above application scenarios will be explained in detail below with reference to the specific accompanying drawings.

[0122] Figure 1This is a schematic diagram illustrating the composition of a voice control system provided in an embodiment of this application. This voice control system can be applied to one or a group of electronic devices 100 as described above. The group of electronic devices includes multiple electronic devices that meet one or more of the following conditions: they are connected to the same wireless access point (such as a WiFi access point); they are logged into the same account; or they are set by the user to be in the same group. The voice control system includes a terminal device 200 and one or more electronic devices 100. The terminal device 200 has a display function, can provide a configuration interface for the user, and can support the user in setting a custom wake-up word and the corresponding usage scenario information. The terminal device can be one of the electronic devices with display function in the group of electronic devices 100, or it can be a device other than the group of electronic devices 200.

[0123] The voice control system may also include a server. This server can be a cloud server or a local server. For example... Figure 1 As shown, the voice control system may also include a server 300. The server is used to perform the operation of determining the wake-up threshold. The server 300 can also be used to manage the wake-up words used by different electronic devices 100 in the voice control system and the actions performed corresponding to the wake-up words.

[0124] For example, the electronic device 100 described in this application embodiment can be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, desktop computer, ultra-mobile personal computer (UMPC), netbook, cellular phone, personal digital assistant (PDA), augmented reality (AR) / virtual reality (VR) device, media player, smart speaker, smartwatch, smart headphones, or smart home device, smart AI device, vehicle terminal, etc. Smart home devices include smart TVs, smart water heaters, smart curtains, smart air conditioners, smart refrigerators, etc. This application embodiment does not impose special limitations on the specific form of the electronic device. Smart AI devices can include, for example, smart robot vacuum cleaners, smart glasses, etc.

[0125] For example, the terminal device 200 described in the embodiments of this application may be a mobile phone, tablet computer, desktop computer, laptop computer, handheld computer, notebook computer, desktop computer, super mobile personal computing device, netbook, cellular phone, AR / VR device, smartwatch, etc.

[0126] refer to Figure 2 The diagram shown is a structural schematic of an electronic device 100 provided in an embodiment of this application.

[0127] The electronic device 100 includes at least a processor 110, a memory 120, and a microphone 130.

[0128] Processor 110 may include one or more processing units, such as application processors (APs), modem processors, graphics processing units (GPUs), image signal processors (ISPs), controllers, video codecs, digital signal processors (DSPs), baseband processors, and / or neural network processing units (NPUs). These different processing units may be independent devices or integrated into one or more processors. The controller can generate operation control signals based on instruction opcodes and timing signals to control instruction fetching and execution.

[0129] The processor 110 may also include a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or that are used repeatedly. If the processor 110 needs to use the instruction or data again, it can retrieve it directly from the memory. This avoids repeated accesses, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0130] In this embodiment, the electronic device 100 may be equipped with a wake-up word (such as "Xiao I Xiao I"). The aforementioned DSP can monitor voice data in real time through the microphone 130 of the electronic device 100. When the DSP detects voice data, it can verify the detected voice data to determine whether it is a wake-up word set in the electronic device 100. If the verification passes, and if the AP of the electronic device 100 is in a sleep state, the DSP can wake up the AP and notify the AP to verify the received voice data again. When the verification passes again, the AP can determine that the voice data matches the wake-up word set in the electronic device 100. The processor 110 may also be equipped with a memory for storing instructions and data. In some embodiments, the memory in the processor 110 is a cache memory. This memory can store instructions or data that the processor 110 has just used or is cyclically used. If the processor 110 needs to use the instruction or data again, it can directly call it from the memory. This avoids repeated access, reduces the waiting time of the processor 110, and thus improves the efficiency of the system.

[0131] Microphone 130, also known as a "microphone" or "voice transducer," is used to convert sound signals into electrical signals. For example, when sending voice data triggers an electronic device to perform certain events, the user can speak by bringing their mouth close to microphone 130, inputting the sound signal into microphone 130. Electronic device 100 may be equipped with at least one microphone 130. In some embodiments, electronic device may be equipped with two microphones 130, which, in addition to collecting sound signals, can also perform noise reduction. In other embodiments, electronic device may be equipped with three, four, or more microphones 130, which can collect sound signals, reduce noise, identify sound sources, and perform directional recording, etc.

[0132] The memory 120 can be used to store computer executable program code, which includes instructions. The processor 110 executes various functional applications and data processing of the electronic device 100 by running the instructions stored in the memory 120. The memory 120 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as a sound playback function), etc. The data storage area may store data created during the use of the electronic device 100 (such as audio data), etc. Furthermore, the memory 120 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc.

[0133] The electronic device 100 may also include a wireless communication module 140. The wireless communication module 140 can provide solutions for wireless communication applications on the electronic device, including wireless local area networks (WLANs) (such as Wireless Fidelity (Wi-Fi) networks), Bluetooth (BT), Global Navigation Satellite System (GNSS), frequency modulation (FM), near-field communication (NFC), and infrared (IR) technologies. In some embodiments of this application, the wireless communication module 140 can interact with other electronic devices. For example, after detecting voice data matching a wake-up word, it can send energy information of the detected voice data to other electronic devices and receive a wake-up indication returned by the electronic device to determine whether a wake-up response is needed. Another example is receiving a content indication sent by the electronic device and executing an event corresponding to a user's voice command based on the content indication.

[0134] It is understood that the structures illustrated in this embodiment do not constitute a specific limitation on the electronic device. The structures of different electronic devices in this application may differ, and these embodiments do not provide examples of each. In other embodiments, the electronic device may include more components than illustrated, or combine certain components, or split certain components, or arrange different components. The illustrated components may be implemented in hardware, software, or a combination of both.

[0135] refer to Figure 3 The diagram shown is a structural schematic of a terminal device 200 provided in an embodiment of this application. As an example, the terminal device 200 also includes a processor 110, a memory 120, and a microphone 130. It may also include a wireless communication mode 140 for the processor 110, memory 120, and microphone 130. The related descriptions of the wireless communication mode 140 have been described above and will not be repeated here. The terminal device 200 also includes a display screen 150.

[0136] Display screen 150 is used to display images, videos, etc. Display screen 150 includes a display panel. The display panel may be a liquid crystal display (LCD), an organic light-emitting diode (OLED), an active-matrix organic light-emitting diode (AMOLED), a flexible light-emitting diode (FLED), a miniature LED, a microLED, a quantum dot light-emitting diode (QLED), etc. In some embodiments, terminal device 200 may include one or N1 displays 150, where N1 is a positive integer greater than 1.

[0137] The terminal device 200 implements display functions through a GPU, a display screen 150, and an application processor. The GPU is a microprocessor for image processing, connected to the display screen 150 and the application processor. The GPU performs mathematical and geometric calculations and is used for graphics rendering. The processor 110 may include one or more GPUs, which execute program instructions to generate or modify display information.

[0138] Furthermore, the terminal device 200 may also include one or more of the following: a charging management module 160, a power management module 141, a battery 142, antenna 1, antenna 2, a mobile communication module 150, a wireless communication module 140, an audio module 170, a speaker 170A, a receiver 170B, a headphone jack 170C, a sensor module 180, buttons 190, a motor 191, an indicator 192, a camera 193, an external memory interface 194, and a subscriber identification module (SIM) card interface 195, a USB interface 196, etc. For example, the sensor module 180 may include a pressure sensor, a gyroscope sensor, a barometric pressure sensor, a magnetic sensor, an accelerometer, a distance sensor, a proximity sensor, a fingerprint sensor, a temperature sensor, a touch sensor, an ambient light sensor, a bone conduction sensor, etc.

[0139] In some embodiments, the processor 110 may include one or more interfaces. Interfaces may include an inter-integrated circuit (I2C) interface, an inter-integrated circuit sound (I2S) interface, a pulse code modulation (PCM) interface, a universal asynchronous receiver / transmitter (UART) interface, a mobile industry processor interface (MIPI), a general-purpose input / output (GPIO) interface, a subscriber identity module (SIM) interface, and / or a universal serial bus (USB) interface, etc.

[0140] The I2C interface is a bidirectional synchronous serial bus, including a serial data line (SDA) and a serial clock line (SCL). In some embodiments, the processor 110 may include multiple I2C buses. The processor 110 can couple to the touch sensor 180K, charger, flash, camera 193, etc., through different I2C bus interfaces. For example, the processor 110 can couple to the touch sensor 180K through the I2C interface, enabling the processor 110 and the touch sensor 180K to communicate through the I2C bus interface, thereby realizing the touch function of the terminal device 200.

[0141] The I2S interface can be used for audio communication. In some embodiments, the processor 110 may include multiple I2S buses. The processor 110 can be coupled to the audio module 170 via the I2S bus to enable communication between the processor 110 and the audio module 170. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 140 via the I2S interface to enable the function of answering phone calls through the headset 200 (e.g., a Bluetooth headset).

[0142] The PCM interface can also be used for audio communication, sampling, quantizing, and encoding analog signals. In some embodiments, the audio module 170 and the wireless communication module 140 can be coupled via the PCM bus interface. In some embodiments, the audio module 170 can also transmit audio signals to the wireless communication module 140 via the PCM interface, enabling the function of answering phone calls through the Bluetooth headset 200. Both the I2S interface and the PCM interface can be used for audio communication.

[0143] The UART interface is a universal serial data bus used for asynchronous communication. This bus can be a bidirectional communication bus. It converts the data to be transmitted between serial and parallel communication. In some embodiments, the UART interface is typically used to connect the processor 110 and the wireless communication module 140. For example, the processor 110 communicates with the Bluetooth module in the wireless communication module 140 via the UART interface to implement Bluetooth functionality. In some embodiments, the audio module 170 can transmit audio signals to the wireless communication module 140 via the UART interface to enable music playback through headphones.

[0144] The MIPI interface can be used to connect the processor 110 to peripheral devices such as the display screen 150 and the camera 193. The MIPI interface includes a camera serial interface (CSI) and a display serial interface (DSI). In some embodiments, the processor 110 and the camera 193 communicate via the CSI interface to enable the shooting function of the terminal device 200. The processor 110 and the display screen 150 communicate via the DSI interface to enable the display function of the terminal device 200.

[0145] The GPIO interface is configurable via software. It can be configured as a control signal or a data signal. In some embodiments, the GPIO interface can be used to connect the processor 110 to a camera 193, a display screen 150, a wireless communication module 140, an audio module 170, a sensor module 180, etc. The GPIO interface can also be configured as an I2C interface, an I2S interface, a UART interface, a MIPI interface, etc.

[0146] USB interface 196 is an interface compliant with the USB standard specification, specifically a Mini USB interface, Micro USB interface, USB Type-C interface, etc. USB interface 196 can be used to connect a charger to charge terminal device 200, and can also be used for data transfer between terminal device 200 and peripheral devices. It can also be used to connect headphones for audio playback. This interface can also be used to connect other electronic devices, such as AR devices.

[0147] It is understood that the interface connection relationships between the modules illustrated in the embodiments of the present invention are merely illustrative and do not constitute a structural limitation on the terminal device 200. In other embodiments of this application, the terminal device 200 may also employ different interface connection methods or combinations of multiple interface connection methods as described in the above embodiments.

[0148] The charging management module 160 receives charging input from a charger. The charger can be a wireless charger or a wired charger. In some wired charging embodiments, the charging management module 160 receives charging input from the wired charger via a USB interface 196. In some wireless charging embodiments, the charging management module 160 receives wireless charging input via the wireless charging coil of the terminal device 200. While charging the battery 142, the charging management module 160 can also supply power to the electronic device via the power management module 141.

[0149] The power management module 141 connects the battery 142, the charging management module 160, and the processor 110. The power management module 141 receives input from the battery 142 and / or the charging management module 160, providing power to the processor 110, internal memory 121, display screen 150, camera 193, and wireless communication module 140, etc. The power management module 141 can also monitor parameters such as battery capacity, battery cycle count, and battery health status (leakage current, impedance). In some other embodiments, the power management module 141 may also be located within the processor 110. In other embodiments, the power management module 141 and the charging management module 160 may be located in the same device.

[0150] The wireless communication function of the terminal device 200 can be implemented through antenna 1, antenna 2, mobile communication module 150, wireless communication module 140, modem processor, and baseband processor.

[0151] Antenna 1 and antenna 2 are used to transmit and receive electromagnetic wave signals. Each antenna in terminal device 200 can be used to cover one or more communication frequency bands. Different antennas can also be multiplexed to improve antenna utilization. For example, antenna 1 can be multiplexed as a diversity antenna for a wireless local area network. In some other embodiments, the antennas can be used in conjunction with a tuning switch.

[0152] The mobile communication module 150 can provide solutions for wireless communication, including 2G / 3G / 4G / 5G, applied to the terminal device 200. The mobile communication module 150 may include at least one filter, switch, power amplifier, low noise amplifier (LNA), etc. The mobile communication module 150 can receive electromagnetic waves via antenna 1, and perform filtering, amplification, and other processing on the received electromagnetic waves before transmitting them to a modem processor for demodulation. The mobile communication module 150 can also amplify the signal modulated by the modem processor and convert it into electromagnetic waves for radiation via antenna 1. In some embodiments, at least some functional modules of the mobile communication module 150 may be housed in the processor 110. In some embodiments, at least some functional modules of the mobile communication module 150 and at least some modules of the processor 110 may be housed in the same device.

[0153] The modem processor may include a modulator and a demodulator. The modulator modulates the low-frequency baseband signal to be transmitted into a mid-to-high frequency signal. The demodulator demodulates the received electromagnetic wave signal into a low-frequency baseband signal. The demodulator then transmits the demodulated low-frequency baseband signal to the baseband processor for processing. After processing by the baseband processor, the low-frequency baseband signal is transmitted to the application processor. The application processor outputs sound signals through an audio device (not limited to speaker 170A, receiver 170B, etc.) or displays images or videos through the display screen 150. In some embodiments, the modem processor may be a separate device. In other embodiments, the modem processor may be independent of the processor 110 and may be housed in the same device as the mobile communication module 150 or other functional modules.

[0154] The wireless communication module 140 can provide solutions for wireless communication applications on the terminal device 200, including wireless local area networks (WLAN) (such as wireless fidelity (Wi-Fi) networks), Bluetooth (BT), global navigation satellite system (GNSS), frequency modulation (FM), near field communication (NFC), and infrared (IR) technologies. The wireless communication module 140 can be one or more devices integrating at least one communication processing module. The wireless communication module 140 receives electromagnetic waves via antenna 2, performs frequency modulation and filtering of the electromagnetic wave signals, and sends the processed signal to processor 110. The wireless communication module 140 can also receive signals to be transmitted from processor 110, perform frequency modulation and amplification, and convert them into electromagnetic waves for radiation via antenna 2. For example, the wireless communication module 140 may include a Bluetooth module, allowing the terminal device 200 to establish wireless connections with other electronic devices. Alternatively, the wireless communication module 140 may include an infrared module, allowing the terminal device 200 to establish wireless connections with other electronic devices.

[0155] In some embodiments, antenna 1 of terminal device 200 is coupled to mobile communication module 150, and antenna 2 is coupled to wireless communication module 140, enabling terminal device 200 to communicate with networks and other devices via wireless communication technology. The wireless communication technology may include Global System for Mobile Communications (GSM), General Packet Radio Service (GPRS), Code Division Multiple Access (CDMA), Wideband Code Division Multiple Access (WCDMA), Time Division Code Division Multiple Access (TD-SCDMA), Long Term Evolution (LTE), BT, GNSS, WLAN, NFC, FM, and / or IR technologies, etc. The GNSS may include the Global Positioning System (GPS), the Global Navigation Satellite System (GLONASS), the BeiDou Navigation Satellite System (BDS), the Quasi-Zenith Satellite System (QZSS), and / or satellite-based augmentation systems (SBAS).

[0156] The terminal device 200 can perform shooting functions through an ISP, camera 193, video codec, GPU, display 150, and application processor.

[0157] The ISP (Image Signal Processor) is used to process data fed back from the camera 193. For example, when taking a picture, the shutter is opened, and light is transmitted through the lens to the camera's photosensitive element. The light signal is converted into an electrical signal, and the camera's photosensitive element transmits the electrical signal to the ISP for processing, transforming it into an image visible to the naked eye. The ISP can also perform algorithmic optimization of image noise, brightness, and skin tone. The ISP can also optimize parameters such as exposure and color temperature of the shooting scene. In some embodiments, the ISP can be set in the camera 193.

[0158] Camera 193 is used to capture still images or videos. An object is projected onto a photosensitive element by generating an optical image through the lens. The photosensitive element can be a charge-coupled device (CCD) or a complementary metal-oxide-semiconductor (CMOS) phototransistor. The photosensitive element converts the light signal into an electrical signal, which is then transmitted to an ISP for conversion into a digital image signal. The ISP outputs the digital image signal to a DSP for processing. The DSP converts the digital image signal into image signals in standard formats such as RGB and YUV. In some embodiments, processor 110 can trigger the camera 193 to start according to a program or instruction in internal memory 121, so that camera 193 acquires at least one image and performs corresponding processing on at least one image according to the program or instruction. In some embodiments, terminal device 200 may include one or N2 cameras 193, where N2 is a positive integer greater than 1.

[0159] A digital signal processor (DSP) is used to process digital signals. Besides digital image signals, it can also process other digital signals. For example, when the terminal device 200 selects a frequency, the DSP can perform Fourier transforms on the frequency energy.

[0160] Video codecs are used to compress or decompress digital video. Terminal device 200 may support one or more video codecs. Thus, terminal device 200 can play or record video in various encoding formats, such as Moving Picture Experts Group (MPEG) 1, MPEG 2, MPEG 3, MPEG 4, etc.

[0161] NPU stands for Neural Network (NN) Computing Processor. By borrowing the structure of biological neural networks, such as the transmission patterns between neurons in the human brain, it can rapidly process input information and continuously learn on its own. NPUs enable intelligent cognitive applications in terminal devices, such as image recognition, facial recognition, speech recognition, and text understanding.

[0162] The external storage interface 194 can be used to connect an external storage card, such as a Micro SD card, to expand the storage capacity of the terminal device 200. The external storage card communicates with the processor 110 through the external storage interface 194 to perform data storage functions. For example, music, video, and other files can be saved on the external storage card.

[0163] Internal memory 121 can be used to store computer executable program code, which includes instructions. Internal memory 121 may include a program storage area and a data storage area. The program storage area may store the operating system, at least one application program required for a function (such as a camera application), etc. The data storage area may store data created during the use of the terminal device 200 (such as images captured by a camera), etc. Furthermore, internal memory 121 may include high-speed random access memory, and may also include non-volatile memory, such as at least one disk storage device, flash memory device, universal flash storage (UFS), etc. Processor 110 executes various functional applications and data processing of the terminal device 200 by running instructions stored in internal memory 121 and / or instructions stored in memory located in the processor.

[0164] Terminal device 200 can implement audio functions, such as music playback and recording, through audio module 170, speaker 170A, receiver 170B, microphone 170C, headphone jack 170D, and application processor.

[0165] The audio module 170 is used to convert digital audio information into analog audio signals for output, and also to convert analog audio input into digital audio signals. The audio module 170 can also be used for encoding and decoding audio signals. In some embodiments, the audio module 170 may be located in the processor 110, or some functional modules of the audio module 170 may be located in the processor 110.

[0166] The speaker 170A, also known as a "loudspeaker," is used to convert audio electrical signals into sound signals. The terminal device 200 can listen to music or make hands-free calls through the speaker 170A.

[0167] The receiver 170B, also known as the "earpiece," is used to convert audio electrical signals into sound signals. When the terminal device 200 receives a telephone call or voice data, the receiver 170B can be brought close to the listener's ear to receive the voice.

[0168] A pressure sensor is used to sense pressure signals and convert them into electrical signals. In some embodiments, the pressure sensor may be located on the display screen 150. There are many types of pressure sensors, such as resistive pressure sensors, inductive pressure sensors, and capacitive pressure sensors. A capacitive pressure sensor may include at least two parallel plates with conductive material. When force is applied to the pressure sensor, the capacitance between the electrodes changes. The terminal device 200 determines the pressure intensity based on the change in capacitance. When a touch operation is applied to the display screen 150, the terminal device 200 detects the intensity of the touch operation based on the pressure sensor. The terminal device 200 may also calculate the touch position based on the detection signal from the pressure sensor. In some embodiments, touch operations applied to the same touch position but with different intensities may correspond to different operation commands. For example, when a touch operation with an intensity less than a first pressure threshold is applied to the SMS application icon, a command to view an SMS message is executed. When a touch operation with an intensity greater than or equal to the first pressure threshold is applied to the SMS application icon, a command to create a new SMS message is executed.

[0169] An accelerometer can detect the magnitude of acceleration of a terminal device 200 in various directions (typically three axes). When the terminal device 200 is stationary, it can detect the magnitude and direction of gravity. It can also be used to identify the posture of electronic devices and is applied to applications such as screen orientation switching and pedometers.

[0170] A distance sensor is used to measure distance. The terminal device 200 can measure distance via infrared or laser. In some embodiments, during a shooting scene, the terminal device 200 can utilize the distance sensor to measure distance for rapid focusing.

[0171] A fingerprint sensor is used to collect fingerprints. Terminal device 200 can use the collected fingerprint characteristics to achieve fingerprint unlocking, accessing application locks, taking photos with fingerprints, answering calls with fingerprints, etc.

[0172] A touch sensor, also known as a "touch device," can be located on the display screen 150. The touch sensor and the display screen 150 together form a touchscreen, also called a "touchscreen." The touch sensor detects touch operations applied to or near it. The touch sensor can transmit the detected touch operation to the application processor to determine the type of touch event. Visual output related to the touch operation can be provided through the display screen 150. In some embodiments, the touch sensor may also be located on the surface of the terminal device 200, in a different position than the display screen 150.

[0173] Buttons 190 include a power button, volume buttons, etc. Buttons 190 can be mechanical buttons or touch-sensitive buttons. Terminal device 200 can receive button input and generate key signal inputs related to user settings and function control of terminal device 200.

[0174] Motor 191 can generate vibration alerts. Motor 191 can be used for incoming call vibration alerts or for touch vibration feedback. For example, different vibration feedback effects can be corresponding to touch operations applied to different applications (such as taking photos, playing audio, etc.). Motor 191 can also correspond to different vibration feedback effects for touch operations applied to different areas of the display screen 150. Different application scenarios (such as time reminders, receiving messages, alarm clocks, games, etc.) can also correspond to different vibration feedback effects. The touch vibration feedback effect can also be customized.

[0175] Indicator 192 can be an indicator light, used to indicate charging status, power changes, or to indicate messages, missed calls, notifications, etc.

[0176] The SIM card interface 195 is used to connect a SIM card. The SIM card can be inserted into or removed from the SIM card interface 195 to make contact with and separate from the terminal device 200. The terminal device 200 can support one or N3 SIM card interfaces, where N3 is a positive integer greater than 1. The SIM card interface 195 can support Nano SIM cards, Micro SIM cards, SIM cards, etc. Multiple cards can be inserted into the same SIM card interface 195 simultaneously. The multiple cards can be of the same or different types. The SIM card interface 195 is also compatible with different types of SIM cards. The SIM card interface 195 is also compatible with external memory cards. The terminal device 200 interacts with the network through the SIM card to realize functions such as calls and data communication. In some embodiments, the terminal device 200 uses an eSIM, i.e., an embedded SIM card. The eSIM card can be embedded in the terminal device 200 and cannot be separated from the terminal device 200.

[0177] The software system of terminal device 200 can adopt a layered architecture, event-driven architecture, microkernel architecture, microservice architecture, or cloud architecture. This embodiment of the invention uses the layered architecture Android system as an example to exemplify the software structure of terminal device 200.

[0178] Figure 4A This is a software structure block diagram of the terminal device 200 according to an embodiment of the present invention.

[0179] A layered architecture divides software into several layers, each with a clear role and function. Layers communicate with each other through software interfaces. In some embodiments, the Android system is divided into four layers, from top to bottom: the application layer, the application framework layer, the Android runtime and system libraries, and the kernel layer. The application layer can include a series of application packages.

[0180] like Figure 4A As shown, the application package may include applications such as camera, gallery, calendar, call, map, navigation, WLAN, Bluetooth, music, video, SMS, and settings.

[0181] The application framework layer provides application programming interfaces (APIs) and a programming framework for applications in the application layer. The application framework layer includes some predefined functions.

[0182] like Figure 4A As shown, the application framework layer may include a window manager, content provider, view system, phone manager, resource manager, notification manager, and voice wake-up engine, etc.

[0183] The window manager is used to manage windowed applications. It can retrieve screen size, determine the presence of a status bar, lock the screen, and capture screenshots, among other things.

[0184] Content providers store and retrieve data, making that data accessible to applications. This data may include videos, images, audio, made and received phone calls, browsing history and bookmarks, phone books, etc.

[0185] A view system includes visual controls, such as controls for displaying text and controls for displaying images. View systems can be used to build applications. A display interface can consist of one or more views. For example, a display interface including a text notification icon could include views for displaying text and views for displaying images.

[0186] The phone manager is used to provide communication functions for terminal device 200. For example, it manages call status (including connection and disconnection).

[0187] The file explorer provides applications with various resources, such as localized strings, icons, images, layout files, video files, and more.

[0188] The notification manager allows applications to display notifications in the status bar. These notifications can be used to deliver informational messages and can disappear automatically after a short pause, requiring no user interaction. For example, the notification manager can be used to notify users of download completion or message alerts. The notification manager can also display notifications as icons or scrolling text in the top status bar, such as notifications from background applications, or as dialog boxes on the screen. Examples include displaying text messages in the status bar, emitting sounds, vibrating the device, and flashing indicator lights.

[0189] The voice wake-up engine is used to implement the voice wake-up service. When the voice information of the surrounding environment includes a wake word, the voice wake-up service can be executed.

[0190] The Android Runtime consists of core libraries and a virtual machine. The Android runtime is responsible for the scheduling and management of the Android system.

[0191] The core library consists of two parts: one part is the functionalities that need to be called by the Java language, and the other part is the Android core library.

[0192] The application layer and application framework layer run in a virtual machine. The virtual machine executes the Java files of the application layer and application framework layer as binary files. The virtual machine is used to perform functions such as object lifecycle management, stack management, thread management, security and exception management, and garbage collection.

[0193] System libraries can include multiple functional modules. For example: surface manager, media libraries, 3D graphics processing libraries (e.g., OpenGL ES), 2D graphics engines (e.g., SGL), etc.

[0194] The Surface Manager is used to manage the display subsystem and provides the blending of 2D and 3D layers for multiple applications.

[0195] The media library supports playback and recording of various common audio and video formats, as well as still image files. It supports multiple audio and video encoding formats, such as MPEG4, H.264, MP3, AAC, AMR, JPG, and PNG.

[0196] The 3D graphics processing library is used to implement 3D graphics drawing, image rendering, compositing, and layer processing.

[0197] A 2D graphics engine is a graphics engine for 2D drawing.

[0198] The kernel layer is the layer between hardware and software. The kernel layer includes at least display drivers, camera drivers, audio drivers, headphone drivers, and sensor drivers.

[0199] In some application scenarios, the application framework layer in terminal device 200 can also deploy Huawei Mobile Services (HMS Core), and a voice wake-up engine can be deployed in HMS Core. See [link to HMS Core]. Figure 4B As shown. The voice wake-up engine is used to implement the voice wake-up service. When the voice information of the surrounding environment includes a wake word, the voice wake-up service can be executed.

[0200] The following example illustrates the workflow of the terminal device 200's software and hardware, using a scenario where a user inputs a custom wake word.

[0201] When the touch sensor receives a touch operation, the corresponding hardware interrupt is sent to the kernel layer. The kernel layer processes the touch operation into a raw input event (including touch coordinates, touch operation timestamp, etc.). The raw input event is stored in the kernel layer. The application framework layer retrieves the raw input event from the kernel layer, identifies the corresponding input box control for inputting a custom wake word, and the user enters the custom wake word into the input box control, thus the input box control receives the custom wake word.

[0202] The following example, using a voice wake-up scenario, illustrates the workflow of the software and hardware of terminal device 200.

[0203] When the voice sensor detects voice data, it transmits the voice data to the voice wake-up engine. The voice wake-up engine determines whether the voice data includes a wake word. If it does, it executes the wake-up operation corresponding to the wake word. For example, if the wake word is included, the corresponding operation is to play music. The voice wake-up engine can trigger the application layer to start the application and play music.

[0204] See Figure 5 The diagram shown is a logical architecture diagram of the voice control system provided in an embodiment of this application. Logically, the voice control system may include: a central management module 501, a human-computer interaction module 502, a device management module 503, a cloud service module 504, a database module 505, and a threshold determination module 506. It should be understood that... Figure 5 The division of units in the illustrated voice control system is merely a logical functional division. In actual implementation, some units can be integrated into a single physical entity, or they can be physically separated. The voice control system can be implemented with some units in software via processing element calls, and others in hardware. The components shown can be implemented in hardware, software, or a combination of both. It is understood that the structure illustrated in this embodiment does not constitute a specific limitation on the voice control system. Different voice control systems in this application may have different modules, and this application does not provide examples of each. In other embodiments, the voice control system may include more or fewer modules than illustrated, or combine some modules, split some modules, or have different module arrangements.

[0205] The central management module 501 is primarily used to manage the collaboration between various modules, enabling users to set custom wake words and determine their wake-up thresholds. As an example, the central management module 501 can be implemented by the server 300 or by the terminal device 200 responsible for management.

[0206] The human-computer interaction module 502 provides a device interaction interface for users, allowing them to set custom wake words and select attributes such as usage mode (private or public), device mode (near-field or far-field), and device collaboration mode. The human-computer interaction module 502 can be implemented by the terminal device 200 responsible for management.

[0207] The device management module 503 is used to uniformly manage electronic device groups, as well as the wake-up words used by different electronic devices and the corresponding actions to be performed after wake-up. After a custom wake-up word is entered through the human-computer interaction module 502, the custom wake-up word can be synchronized to different electronic devices. For example, the custom wake-up word can be added to all electronic devices registered in the voice control system or to a specified electronic device. The device management module 503 can be implemented by the server 300 or by the terminal device 200 responsible for management.

[0208] The cloud service module 504 is used to uniformly manage custom wake words and wake-up thresholds for different electronic devices. Simultaneously, the cloud service module 504 can retrieve stored custom wake words and wake-up thresholds for different users. When a user's set wake word exists in the cloud, there is no need to execute the wake-up threshold determination process; the wake-up threshold for that custom wake word can be directly synchronized from the cloud.

[0209] The 504 error in the cloud service module can be resolved using a cloud-based server. Alternatively, if a cloud-based server is not deployed, it can be resolved using a local server.

[0210] Database module 505 is used to store different custom wake words and corresponding wake-up thresholds under different attributes, as well as related statistical data generated in the process of determining the wake-up threshold of the custom wake words.

[0211] The database module can be deployed in the cloud, locally, or with separate databases deployed in both locations. The local database stores the wake word currently used by the electronic device, the wake-up threshold corresponding to the wake word, the action (or command) executed after waking up, and related statistical data. The cloud database stores custom wake words, wake-up thresholds, the action executed after waking up for each wake word, and related statistical data for all electronic devices registered under the user account.

[0212] The threshold determination module 506 is used for wake-up word recognition. Based on the user-defined wake-up word and its corresponding attributes, it synthesizes wake-up word audio data of different styles corresponding to the attributes, and calculates or adjusts the wake-up threshold based on the different styles of wake-up word audio data. The threshold determination module 506 may further include a wake-up module 506A, a speech synthesis module 506B, and an adaptive module 506C.

[0213] The wake-up module 506A is used for wake-up word recognition. When a wake-up word is detected in the speech, it provides feedback on the wake-up status and outputs the probability score of the wake-up word, the probability score of each modeling unit, and forced alignment information.

[0214] The speech synthesis module 506B is used to synthesize wake word audio data of different styles according to the wake word and corresponding attributes set by the user.

[0215] The adaptive module 506C is used to calculate and adjust the wake-up threshold of the wake-up word using the results obtained by the wake-up module 506A and the data synthesized by the speech synthesis module.

[0216] The voice control system may also include a voice recognition module 507. The voice recognition module 507 is deployed in each electronic device 100. The voice recognition module 507 receives a wake-up threshold and a wake-up word. Based on the wake-up word and the wake-up threshold, it recognizes the wake-up word spoken by the user and triggers the execution of the action command corresponding to the wake-up word.

[0217] In the following embodiments, the device for determining the execution threshold is taken as an example as the server. This server can be a cloud server, a local server, or an application server. Users input custom wake-up words and their attributes through the display interface provided by the terminal device. The attributes of the custom wake-up word can include one or more of the following: the usage scenario (including usage mode, device mode), multi-device collaboration mode, and the corresponding action command. Optionally, the attributes of the custom wake-up word can also include location area information, time period information, or local area network information used by the custom wake-up word.

[0218] The terminal device sends a custom wake-up word and its attributes to the server. The server then determines the wake-up threshold for the custom wake-up word based on its usage scenario and sends the custom wake-up word, its wake-up threshold, and the action command to be executed after activation to one or more corresponding electronic devices. In one example, the server may also send the usage scenario information of the custom wake-up word to one or more electronic devices. In another example, the server may also send the location area information, time period information, or local area network information of the user-set custom wake-up word to one or more electronic devices. Thus, one or more electronic devices determine whether the user's spoken voice can wake the electronic device to execute the corresponding action based on the custom wake-up word and the wake-up threshold.

[0219] The audio control method for the first possible application scenario will be described in detail below with reference to specific embodiments.

[0220] The embodiments of this application can be applied to applications with control functions, such as intelligent management, smart assistants, smart voice, system settings, and other applications.

[0221] For example, the terminal device displays the main interface on screen 150, such as... Figure 6 The user interface 600 shown is described above. The user interface 600 may include a status bar 601, a hideable navigation bar 602, and icons for multiple applications (e.g., a settings icon 603). The status bar 601 may include a mobile network identifier (e.g., 5G), time, and remaining battery power. It is understood that in some other embodiments, the status bar 601 may also include Bluetooth icons, WiFi icons, external device icons, etc. It is also understood that in some other embodiments, the user interface 600 may include a quick application bar. This quick application bar may also be called a Dock bar, a frequently used application bar, etc. The quick application bar may include icons for frequently used applications. When the terminal device detects a user's finger (or stylus, etc.) touching an application's icon, in response to the touch operation, the terminal device launches the application and displays its user interface on the display screen 150.

[0222] For example, if the terminal device detects a touch operation on the settings icon 603, it will display the system settings interface on the display screen 150 in response to the touch operation. For instance, the system settings interface may be as follows: Figure 7A The user interface 710 is shown. User interface 710 includes wake-up settings 701. In other embodiments, user interface 710 may also include multiple settings options for other functions, such as login account, mobile network, sound, and vibration. In response to a user's operation on wake-up settings 701, the terminal device displays user interface 720 on display screen 150 to participate in... Figure 7B and Figure 7C As shown. The user interface 720, also known as the wake-up settings interface, may include voice wake-up 721, which is used to configure the on / off state of the voice wake-up function of the terminal device. The user interface 720 may also include one or more of the following: adding a custom wake-up word 722, wake-up word management 723, or device management 724.

[0223] For example, in response to the voice wake-up operation 721, the terminal device displays a user interface 730 on the display screen 150. The user interface 730 includes a virtual button 731. The terminal device enables the voice wake-up function in response to the user setting the virtual button 731 to ON. As another example, the terminal device disables the voice wake-up function in response to the user setting the virtual button 771 to OFF.

[0224] In some embodiments, the terminal device also supports changing the wake word of the terminal device. For example, see... Figure 7B and Figure 7C As shown, the user interface 730 may include a wake-word change 732, and the terminal device displays the user interface 740 in response to the user's operation on the wake-word change 732. The user interface 740 includes a wake-word that the user can select to wake up the terminal device. Figure 7B Taking "Xiao A Xiao A", "Da Bao Da Bao", and "Xiao Hua Xiao Hua" as examples, different wake words correspond to selectable virtual buttons 741. In some embodiments, only one wake word can be selected by the user. The user can select one of "Xiao A Xiao A", "Da Bao Da Bao", or "Xiao Hua Xiao Hua" as the wake word for the terminal device, see [link to documentation]. Figure 7B As shown. In other embodiments, users can select multiple wake words, and speaking any one of these wake words will wake up the terminal device. Figure 7C For example, this example supports users selecting multiple wake words.

[0225] See Figure 7A In version 722, adding a custom wake-up word allows adding a custom wake-up word to any electronic device in the voice control system. This electronic device can be the terminal device itself, or any other electronic device besides the terminal device. See also... Figure 8A As shown, the terminal device responds to the user's operation of adding a custom wake word 722 (such as a touch operation or a hover selection operation), and displays a user interface 810. In some embodiments, the display interface 810 includes a text input box 811. See also Figure 8A As shown. In other embodiments, the display interface 810 may also indicate the input of a custom wake word by voice, including the option to input a wake word by voice 812.

[0226] In one possible example, the display interface 810 may also include selection controls for configuring usage scenario information (e.g., usage mode and / or device mode) for custom wake words. In another possible example, if the display interface 810 does not include a usage mode selection control, then the default is private mode. If the display interface 810 does not include a device mode selection control, then the default is near-field mode. Figure 8A Taking the display interface 810 as an example, it includes usage mode 813 and device mode 814. Optionally, the display interface 810 may also include user instructions to prompt the user on how to select a mode and how to enter a custom wake word.

[0227] In some embodiments, the terminal device displays a user interface 820 in response to a user's operation on the text input box 811. The user interface 820 includes a virtual keyboard. See [link to relevant documentation]. Figure 8B As shown, in response to the user's input of a custom wake word via the virtual keyboard, the user interface 830 is displayed to obtain the custom wake word. For example, the user inputs "Xiao Ke Xiao Ke" into the text input box 811 via the virtual keyboard.

[0228] In other embodiments, the terminal device displays a user interface 840 in response to a user's operation on the voice input wake-up word 812 option, see [link to previous embodiment]. Figure 8C As shown. In one example, after displaying the user interface 840, a prompt message is played through a speaker or a prompt box is displayed to prompt the user to say a custom wake-up word N times in a quiet environment, where N is an integer greater than 1. The terminal device, through its voice recognition function, recognizes that the user has said the custom wake-up word N times, and then converts the spoken wake-up word into text. In one example, in response to the user clicking the "Start" button, the terminal device prompts the user to start reading the custom wake-up word through a speaker. After recognizing that the user has read the custom wake-up word N times and obtaining the text of the wake-up word, the terminal device displays the text of the wake-up word on the display interface 840. See [link to relevant documentation]. Figure 8C As shown. In another example, in response to the user clicking the "Start" button, the terminal device prompts the user to start reading a custom wake-up word through the speaker. After recognizing the user reading the custom wake-up word once, the terminal device obtains the text of the wake-up word, displays the text of the wake-up word on the display interface 840, and prompts the user to read the custom wake-up word again. If the recognized custom wake-up word is the same as the previously recognized custom wake-up word read by the user, the terminal device accumulates the number of times the user reads the word; otherwise, it does not record the reading and prompts the user to read it again, until it is determined that the user has read N valid custom wake-up words.

[0229] In one possible example, the user can further select the usage mode of the custom wake word. In some embodiments, a private mode is used as an example. In private mode, it is for private use only, such as on a mobile phone, tablet, or computer. In this private mode, wake-up threshold optimization is only performed for the user. In private mode, voice settings requirements must be met, meaning the user needs to input voice data for the custom wake word. When the terminal device determines that the user has input the custom wake word via text, it determines that the voice settings requirements for private mode are not met and prompts the private user to input voice data for the custom wake word.

[0230] In some embodiments, combined with Figure 8B In response to the user's selection of private mode in usage mode 813, the terminal device displays user interface 850 when it determines that the user has entered a custom wake word in the manner described herein. (See also...) Figure 8DAs shown. The user interface 850 may include prompts for a custom wake-up word to be read aloud to the user. See also Figure 8D As shown, taking "Xiao Ke Xiao Ke" as an example, a custom wake-up word that the user needs to read aloud. After displaying the user interface 850, a prompt message is played through the speaker, prompting the user to say the custom wake-up word "Xiao Ke Xiao Ke" N times in a quiet environment, where N is an integer greater than 1. The terminal device, through its voice recognition function, recognizes that the user has said the custom wake-up word N times and then confirms that the voice recording is complete. In one example, in response to the user clicking the "Start" button, the terminal device prompts the user to start reading the custom wake-up word through the speaker. After recognizing that the user has read the custom wake-up word N times, the device obtains the text of the wake-up word and displays the text of the wake-up word on the display interface 850. See [link to example]. Figure 8D As shown. In another example, in response to the user clicking the "Start" button, the terminal device prompts the user to start reading a custom wake-up word through the speaker. After recognizing the user reading the custom wake-up word once, the terminal device obtains the text of the wake-up word, displays the text of the wake-up word on the display interface 850, and prompts the user to read the custom wake-up word again. If the recognized custom wake-up word is the same as the custom wake-up word entered in the text, the terminal device accumulates the number of times the user reads the word; otherwise, it does not record the reading and prompts the user to read it again, until it is determined that the user has read N valid custom wake-up words.

[0231] In other embodiments, combined with Figure 8C In response to the user's selection of private mode in usage mode 813, the terminal device, upon determining that the user has entered a custom wake-up word via voice input, checks whether the entered custom wake-up word meets the voice settings requirements of private mode. If not, it prompts the private user to enter the voice data for the custom wake-up word. For example, if the private user currently registered in the voice control system is different from the user who entered the custom wake-up word via voice input, the voice settings requirements of private mode are not met. For instance, if the private user registered in the current voice control system is user A, and the user entering the custom wake-up word via voice input is user B, then the voice settings requirements of private mode are not met. Combined with... Figure 8C In response to the user's selection of a private mode in usage mode 813, if the terminal device determines that the voice data for the custom wake-up word does not meet the voice settings requirements of the private mode, it displays user interface 850. (See also...) Figure 8DAs shown, as an example, determining whether the voice data of a custom wake-up word input meets the semantic settings requirements of private mode can be achieved by performing spectral analysis on the voice data of the custom wake-up word input and the saved voice data of private user A to determine whether the voice data of the custom wake-up word input is indeed the voice data of private user A. For example, voiceprint recognition can be used to determine whether the voice data of the custom wake-up word input is indeed the voice data of private user A.

[0232] In some other embodiments, combined with Figure 8B When the terminal device responds to the user's operation of selecting public mode in usage mode 813, the selection control in public mode is selected, and the display format is as follows. Figure 8E The display interface shown.

[0233] In one possible example, the user can also select the device mode for the custom wake word, choosing either near-field or far-field mode as needed.

[0234] In some embodiments, the terminal device sends the voice data of the user-inputted custom wake-up word, the selected usage mode (private mode or public mode), and the device mode (near-field mode or far-field mode) to the server. The server then generates different types of synthesized speech datasets corresponding to the user-defined custom wake-up word and mode information (selected mode and / or device mode), and further confirms the wake-up threshold based on the synthesized speech dataset. In other embodiments, the terminal device determines the wake-up threshold based on the voice data of the user-inputted custom wake-up word, the selected usage mode, and the device mode.

[0235] In other embodiments, users can customize the location, local area network, or time period used for the wake word according to their needs. For example, different locations can correspond to different usage modes. Similarly, different local area networks can correspond to different usage modes. And different time periods can correspond to different usage modes.

[0236] For example, the location and usage scenario can be configured for the same or different wake words. For instance, users can use public or far-field mode at home, and private or near-field mode outside of home.

[0237] For example, location can be obtained through a location sensor in an electronic device, which can be a global positioning system (GPS).

[0238] For example, configurations can be made for the local area network used with the same or different wake words, as well as the usage scenarios within that local area network. For instance, when electronic devices are connected to a home LAN, they can be configured in public mode or far-field mode. When connected to a company LAN, they can be configured in private mode or near-field mode.

[0239] For example, you can configure the time period and usage scenario for the same or different wake words. For instance, you can configure near-field mode at night and remote mode during the day.

[0240] For example, you can configure the time period and local area network used for the same or different wake words. For instance, when connecting to your home LAN at night, you can configure far-field mode and / or public mode, and in other cases, you can configure near-field mode and / or private mode.

[0241] As an example, see Figure 8F The diagram shows one possible configuration location. Figure 8E This is merely an example and does not impose specific limitations on the selection method or interface display format. For instance, an exclusive selection method could be used, excluding usage scenarios for configurations in other locations besides a particular one. Alternatively, an inclusive selection method could be used, selecting usage scenarios for configurations corresponding to a specific location.

[0242] Furthermore, when the server receives the usage scenario information corresponding to the custom wake word and the user-input location area information, time period information, or local area network information, it can configure the electronic device that the custom wake word applies to. The electronic device can associate and store the usage scenario information, location area information, time period information, local area network information, custom wake word, and wake-up threshold.

[0243] For example, time period information can be a certain time period of a day, or a certain day or several days of a month, etc. Local area network information can include service set identifier (SSID), or other information used to identify the local area network.

[0244] See Figure 9 The diagram illustrates the process for determining the wake-up threshold. The wake-up threshold can be determined by the terminal device or by the server. Figure 9 Taking the server as an example, the way terminal devices determine the wake-up threshold is similar to the way servers determine the wake-up threshold; see the section on how servers determine wake-up thresholds.

[0245] 901. The server receives voice data of a custom wake-up word from the terminal device, along with usage scenario information for the custom wake-up word. Usage scenario information includes whether the usage mode is private or public. Device mode may also be included in the usage scenario information.

[0246] 902. The server generates a synthesized speech dataset based on the voice and / or text information of the custom wake word, as well as the usage scenario information of the custom wake word.

[0247] In one example, in private mode, based on the voice data of the private user's custom wake word, a batch of wake-up data in near-field mode that imitates the user's voice is synthesized using voice cloning technology to obtain a synthetic voice dataset. The synthetic voice dataset includes the voice data of the private user's custom wake word and the synthetic voice data obtained using voice cloning technology.

[0248] When the device mode in the usage scenario information is far-field mode, the server converts the synthesized speech dataset in near-field mode into a synthesized speech dataset in far-field mode. For example, the synthesized speech dataset can be convolved with the impact response of a specific room to obtain a synthesized speech dataset in far-field mode.

[0249] In another example, in public mode, a batch of diverse wake-up data mimicking different speaker styles is generated based on the speech data of a custom wake-up word, resulting in a synthetic speech dataset. To ensure data diversity, information such as tone, emotion, and intonation from different speakers can be cloned, and wake-up data with different noise, reverberation, speech rate, or volume can be generated through data augmentation. The synthetic speech dataset includes the speech data of the custom wake-up word and the generated synthetic speech data mimicking different speaker styles.

[0250] 903. When the server determines that the speech energy of the synthesized speech dataset (synthetic speech dataset in near-field mode or speech synthesis dataset in far-field mode) is greater than a preset threshold, wake word detection is performed based on the synthesized speech dataset.

[0251] If the speech energy in the synthesized speech dataset is less than or equal to a preset threshold, wake word detection is not performed. The server determines that if the speech energy is less than or equal to the preset threshold, it indicates that the synthesized speech dataset may not contain data related to wake words. Therefore, before performing wake word detection, it is necessary to determine whether the speech energy meets the requirements to prevent resource waste and increased power consumption due to the failure to detect wake words during the detection process.

[0252] Step 903 is an optional step, and it is also possible to omit step 903.

[0253] 904. The server extracts speech feature information from each basic computational unit included in each synthesized speech data point in the synthesized speech dataset. For example, the basic computational unit can be a time frame, or other defined computational units. The following examples will use a time frame as the basic computational unit.

[0254] 905, the server determines the acoustic posterior probability of the modeling unit corresponding to the M basic computing units based on the speech feature data of each basic computing unit; the modeling unit corresponding to the M basic computing units is the modeling unit included in the first wake-up word.

[0255] Specifically, the server determines the modeling unit S for each synthesized speech data based on the wake-up acoustic model and the speech feature data of each time frame. j The acoustic posterior probability.

[0256] For example, time frame 1 corresponds to an acoustic posterior probability for each modeling unit included in the predefined wake word. If the predefined wake word includes N1 modeling units, then time frame 1 corresponds to an acoustic posterior probability for each of the N1 modeling units.

[0257] The wake-up acoustic model can employ machine learning models, such as deep neural networks (DNN), convolutional neural networks (CNN), or recurrent neural networks (RNN). Alternatively, it can use logistic regression, support vector machines (SVM), Naive Bayes (NB), k-nearest neighbor (KNN), decision tree (DT), or ensemble models, among others.

[0258] In this embodiment, the modeling unit can refer to content such as phonemes, syllables, or Chinese characters. For example, in "xiaoA, xiao A", "xiao" can be a modeling unit, "A" can be a modeling unit, and so on, which will not be elaborated further here. Alternatively, in another implementation, in "xiao", "X" can be a modeling unit, "i" can also be a modeling unit, and so on, which will not be elaborated further here.

[0259] One or more modeling units constitute a decoding path. For example, a decoding path may include “xiao, A, xiao, A”, or “x, i, a, o, A”, etc.

[0260] Furthermore, forced alignment information is obtained by performing Viterbi search on the acoustic posterior probabilities of each modeling unit in the synthesized speech data corresponding to each time frame, namely the correspondence between time frames and modeling units, and the acoustic posterior probability of the modeling unit corresponding to the time frame.

[0261] Optionally, when performing a Viterbi search by mapping the acoustic posterior probabilities of each modeling unit in the synthesized speech data to each time frame, it can be first determined whether a wake-up word exists in the synthesized speech data. If no wake-up word exists, the synthesized speech data can be deleted. For example, noisy time frames and blank time frames can also be deleted during the Viterbi search. If the synthesized speech data includes a predefined wake-up word, the acoustic posterior probability of the modeling unit corresponding to the time frame (e.g., 10ms) in the decoding path corresponding to the predefined wake-up word can be obtained, thus obtaining the forced alignment information of the modeling unit. That is, the correspondence between time frames and modeling units.

[0262] For example, if the synthesized speech data includes 100 time frames, each time frame corresponds to N modeling units, each with an acoustic posterior probability. When a wake word is determined, the acoustic posterior probability of the modeling unit corresponding to the wake word, as well as the time frame in which the modeling unit is located, is determined. For example, if the decoding path is xiaoAxiaoA, then from the 100 time frames, the time frames that are decoded into modeling units x, i, a, o, and A, and their corresponding acoustic posterior probabilities are determined to obtain forced alignment information. For example, time frame 1 is decoded into modeling unit x, and the acoustic posterior probability of time frame 1 corresponding to the decoded modeling unit x is P1.

[0263] For example, the number of time frames of the modeling units decoded into the predefined wake word in the k-th synthesized speech data is m. k For example, if the predefined wake word includes two modeling units, modeling unit 1 and modeling unit 2, then m k One portion of the time frames is decoded as modeling unit 1, and the other portion is decoded as modeling unit 2.

[0264] If the speech synthesis dataset contains Z synthesized speech data, then...

[0265] 906, obtain the M basic computational units corresponding to the j-th modeling unit S. j The average of the acoustic posterior probabilities; j takes all positive integers less than or equal to N1.

[0266] For example, if the effective time segment of the synthesized speech dataset includes N1 modeling units (i.e., the number of modeling units included in the decoding path of the custom wake word), then for a given modeling unit, the effective time segment in the synthesized speech dataset is determined to include the segment decoded into that modeling unit S. jNumber of time frames d j and d j The corresponding modeling unit S in each time frame j The acoustic posterior probability is represented as P(S) j |X i ), thereby obtaining the average value.

[0267] 907. Obtain the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units. The maximum value can be obtained as follows:

[0268] 908. The probability compensation value of the j-th modeling unit is obtained based on the average value and the maximum value of the acoustic posterior probability of the j-th modeling unit.

[0269] For example, modeling unit S j The probability compensation value is: The modeling unit S after compensation j The corrected acoustic posterior probability is P'(S j )=P(S j |X i )+P(S j ).

[0270] 907, the server uses the compensated acoustic posterior probability P'(S) of each modeling unit. j Calculate the overall probability score of the wake word in each synthesized speech data in the decoded synthesized speech dataset. (t is the duration of the wake word). The average likelihood score is obtained by averaging the scores of all synthesized speech data in the synthesized speech dataset. (n is the number of wake words, i.e. the number of synthesized speech data in the synthesized speech dataset), which is used as the new wake-up threshold (or wake-up threshold).

[0271] Alternatively, the likelihood score can be fine-tuned F = f avg +△ serves as the new wake-up threshold. △ can be adjusted based on the actual task; to improve the wake-up rate, △ can be set to a small negative value, while to reduce the false wake-up rate, △ can be set to a small positive value. Optionally, this △ can be a set value derived from experience and configured in the server. The setting of the △ value can be determined by the wake-up sensitivity adjusted by the user of the terminal device; for example, different wake-up sensitivities correspond to different △ values.

[0272] As an example, wake-up settings may include controls for wake-up sensitivity. As an example, combined with... Figure 7B For the user interface 720 shown in the wake-up settings, see [link to user interface]. Figure 10As shown, in response to the user's operation on voice wake-up 721, the terminal device displays user interface 1010, which includes a wake-up sensitivity setting control 1001. Figure 10 In this embodiment, wake-up sensitivity is divided into three levels: low, medium, and high. Different levels correspond to different Δ values; the higher the wake-up sensitivity, the smaller the Δ value. It should be noted that this application does not limit the level division of wake-up sensitivity; more or fewer levels can be used. When the terminal device obtains the wake-up sensitivity level selected by the user, it sends the wake-up sensitivity level to the server. The server then determines the wake-up threshold based on the Δ value corresponding to each wake-up sensitivity level.

[0273] 908. The server sends the new wake-up threshold to the electronic device affected by the custom wake-up word, such as the terminal device, which can then use the wake-up threshold to perform a wake-up operation.

[0274] In some embodiments, when sending a custom wake-up word and the wake-up threshold corresponding to the custom wake-up word, the server may also send the probability compensation values ​​of the N1 modeling units included in the custom wake-up word to the electronic device affected by the custom wake-up word. In the following description, the example is that the server sends the wake-up threshold and the probability compensation values ​​of the modeling units included in the custom wake-up word to the electronic device. The set of probability compensation values ​​includes the probability compensation values ​​of the N1 modeling units included in the custom wake-up word under the usage scenario indicated by the usage scenario information. The probability compensation value of the first modeling unit is used to compensate the acoustic posterior probability of the first modeling unit in the custom wake-up word when determining the acoustic posterior probability of the first modeling unit included in the custom wake-up word in the voice information. The first modeling unit is any one of the N1 modeling units. For example, the custom wake-up word includes xiao, A, xiao, A; the modeling units in the custom wake-up word include xiaoAxiaoA respectively; taking the first modeling unit as x as an example, after the electronic device determines the acoustic posterior probability of x through the acoustic probability model, the probability compensation value of x is used to compensate the acoustic posterior probability of x determined through the acoustic probability model, as the acoustic posterior probability of the modeling unit x. Then, by determining the acoustic posterior probability of each modeling unit, it is determined whether the probability of receiving voice information containing a custom wake-up word reaches the wake-up threshold corresponding to the custom wake-up word, and thus decides whether to wake up the electronic device.

[0275] In other embodiments, when the server sends the custom wake word and the wake-up threshold corresponding to the custom wake word, the server can also send the usage scenario information corresponding to the custom wake word to the electronic device to which the custom wake word is applied.

[0276] In other embodiments, when the server sends a custom wake-up word and the wake-up threshold corresponding to the custom wake-up word, the server may also send the usage scenario information corresponding to the custom wake-up word, as well as the usage location area information, usage time period information, or local area network information of the custom wake-up word to the electronic device that the custom wake-up word is applied to.

[0277] The audio control method for the second possible application scenario will be described in detail below with reference to specific embodiments.

[0278] In the second possible application scenario, the terminal device sets a custom wake word according to user needs. This custom wake word can be applied to one or more electronic devices.

[0279] See one possible example. Figure 11 The diagram shown is a schematic of a possible user interface 1110 for adding a custom wake word. In some embodiments, it combines... Figure 7B The terminal device can display a user interface 1110 in response to the operation of adding a custom wake word 722. The user interface 1110 may include a text box 1111 for inputting the custom wake word and a control 1112 for user voice input of the custom wake word. The user interface 1110 includes a multi-device collaboration selection control 1113. The multi-device collaboration selection control 1113 is used to select the electronic device to which the custom wake word applies, or to select the electronic device to which the custom wake word is synchronized. Optionally, the user interface 1110 may also include a usage mode 1114 and / or a device mode 1115.

[0280] The input method for the custom wake word text box 1111 and Figure 8B The input method for text box 811 is similar; please refer to [link / reference]. Figure 8B Instructions for text box input are not detailed here. Regarding voice input with a custom wake word, the process is similar to... Figure 8C For information on the input methods for custom wake words in Chinese voice input, please refer to [link / reference]. Figure 8C The instructions for custom wake-up words for voice input are not repeated here. The selection method for private mode and... Figure 8D The method for selecting usage modes is similar; please refer to [link / reference]. Figure 8D The explanation of choosing the private mode is omitted here. The explanation of choosing the public mode is as follows... Figure 8E The usage mode selection method type can be... Figure 8E The selection of the public mode is explained below, and will not be repeated here. Additionally, for explanations regarding the selection of the device mode, please refer to the relevant explanations for the device mode in the first possible application scenario, which will not be repeated here.

[0281] The following section focuses on the selection control 1113 for multi-device collaboration.

[0282] In some embodiments, the usage mode is set to private mode as an example. For instance, if a user inputs a custom wake-up phrase via voice as "Xiao A, Xiao A," selects private mode as the usage mode, and near-field mode as the device mode, see [link to documentation]. Figure 12A As shown. The terminal device responds to the user's selection of the multi-device collaboration option. Under the multi-device collaboration option, users can then choose to synchronize to all devices or synchronize to a specified device. When the multi-device collaboration option is not selected, the options to synchronize to all devices and synchronize to a specified device cannot be selected. If synchronize to all devices is selected, the terminal device detects the user's operation on the synchronize to all devices selection control and determines that the custom wake word will be synchronized across a group of electronic devices in the voice control system. See [link to relevant documentation]. Figure 12A As shown. In other embodiments, see [reference needed]. Figure 12B As shown, the terminal device responds to the user's selection of the multi-device collaboration option. Under the multi-device collaboration option, the user can then choose to synchronize to all devices or to a specified device. If synchronization to a specified device is selected, the terminal device detects the user's operation on the selection control for synchronizing to the specified device, and the user can select one or more of the specified devices as needed. Figure 12B This example uses only two electronic devices, electronic device 1 and electronic device 2, to illustrate the ability to select either electronic device 1 or electronic device 2. Specifically, the terminal device detects the user's selection action within the options on electronic device 1, for example, see [link to example]. Figure 12B The display format shown indicates that synchronization with the specified device includes electronic device 1. The terminal device detects the user's selection operation in the options of electronic device 2, for example, see [link to example]. Figure 12B The display format shown indicates that synchronization with the specified device also includes electronic device 2. Therefore, it is determined that the custom wake-up word is synchronized in both electronic device 1 and electronic device 2 within the voice control system. See [link / reference]. Figure 12B As shown.

[0283] In one example, if the user does not select a specific device to sync to, the current terminal device can be used by default. The terminal device sends the user's custom wake-up word voice information (and / or custom wake-up word text) and the user's selected mode information (including usage scenario information and electronic device information where the custom wake-up word applies) to the server. Usage scenario information includes one or more of the usage mode and device mode.

[0284] Furthermore, the server determines a wake-up threshold for the user's custom wake-up word based on the user's voice information (and / or text of the custom wake-up word) and the user's selected usage scenario information, and sends it to the terminal device. In some embodiments, the server may also send the probability compensation value of the modeling unit included in the custom wake-up word determined for the terminal device and the wake-up threshold together to the terminal device.

[0285] In another example, when a user selects to synchronize to a designated device, the terminal device sends the user's custom wake-up word's voice information (and / or the text of the custom wake-up word) and the user's selected mode information (including usage scenario information and information about the electronic device to which the custom wake-up word applies) to the server. The usage scenario information includes one or more of the usage mode or device mode. Further, the server determines a wake-up threshold for the terminal device's custom wake-up word based on the user's custom wake-up word's voice information (and / or the text of the custom wake-up word) and the user's selected usage scenario information (which may also include probability compensation values ​​for the modeling units included in the custom wake-up word), and sends it to the designated device.

[0286] In another example, when a user selects to synchronize to multiple specified devices, the terminal device sends the user's custom wake-up word's voice information (and / or custom wake-up word's text) and the user's selected mode information (including usage scenario information and electronic device information where the custom wake-up word applies) to the server. The usage scenario information includes one or more of the usage mode or device mode. Further, the server determines wake-up thresholds (which may also include probability compensation values ​​for the modeling units included in the custom wake-up word) for the custom wake-up words of the multiple specified devices based on the user's custom wake-up word's voice information (and / or custom wake-up word's text) and the user's selected usage scenario information, and sends these thresholds to the multiple specified devices.

[0287] In another example, when a user chooses to sync to all electronic devices or a specific electronic device, the terminal device can determine the usage mode and / or device mode of each electronic device based on its device type. For example, the association between device type, usage mode, and device mode can be seen in Table 1. In this example, the usage mode and device mode selected by the user for the custom wake word may have no effect.

[0288] Table 1

[0289] Equipment type Usage Mode Device mode cell phone Private mode Near Field Mode flat Private mode Near Field Mode computer Private mode Far-field mode Smart TV Common mode Far-field mode smart speaker Common mode Far-field mode Smart home appliances Common mode Far-field mode Vehicle-mounted equipment Common mode Near Field Mode

[0290] In another example, when a user chooses to synchronize to all electronic devices or a specific electronic device, the user of the terminal device does not need to configure the device mode and usage mode of the specified electronic device. The terminal device sends the user's custom wake-up word voice information (and / or custom wake-up word text) and the user-selected multi-device collaboration information to the server. The user-selected multi-device collaboration information includes the identifier of the specified device to be synchronized or an indication to synchronize to all devices. The server determines the usage mode and / or device mode of each electronic device based on the device type of the synchronized electronic device. Further, the server determines the wake-up threshold (which may also include the probability compensation value of the modeling unit included in the custom wake-up word) for each electronic device based on the user's custom wake-up word voice information (and / or custom wake-up word text) and usage scenario information, and sends the custom wake-up word and the wake-up threshold (which may also include the probability compensation value of the modeling unit included in the custom wake-up word) determined separately for each electronic device to the corresponding electronic device according to the multi-device collaboration information.

[0291] For example, if a specified electronic device includes a tablet, and the tablet's usage mode is private mode and device mode is proximity mode, then the server determines the wake-up threshold and probability compensation value of the modeling unit included in the custom wake-up word for the tablet based on the private mode, proximity mode, the private user's voice information, and the custom wake-up word. Then, the server sends the custom wake-up word and its wake-up threshold to the tablet. As another example, if a specified electronic device includes a smart TV, and the smart TV's usage mode is public mode and device mode is far-field mode, then the server determines the wake-up threshold and probability compensation value of the modeling unit included in the custom wake-up word for the smart TV based on the private mode, proximity mode, and the text (or voice data) of the custom wake-up word. Then, the server sends the custom wake-up word, its wake-up threshold, and the probability compensation value of the modeling unit included in the custom wake-up word to the smart TV. For example, if a specified electronic device includes headphones, and the headphones are used in a private mode and the device mode is near-field mode, then the server determines the wake-up threshold of the custom wake-up word and the probability compensation value of the modeling unit included in the custom wake-up word for the headphones based on the voice data of the private mode, near-field mode, and the custom wake-up word. Then, the server sends the custom wake-up word, its wake-up threshold, and the probability compensation value of the modeling unit included in the custom wake-up word to the headphones. Similarly, if a specified electronic device includes an in-vehicle terminal, and the in-vehicle terminal is used in a private mode and the device mode is near-field mode, then the server determines the wake-up threshold of the custom wake-up word and the probability compensation value of the modeling unit included in the custom wake-up word for the headphones based on the voice data of the private mode, near-field mode, and the custom wake-up word. Then, the server sends the custom wake-up word, its wake-up threshold, and the probability compensation value of the modeling unit included in the custom wake-up word to the in-vehicle terminal.

[0292] In some embodiments, users configure the usage mode, device mode, and actions to be performed when waking up the electronic device using a custom wake word, according to their needs.

[0293] In one example, combining Figure 12B In response to the user selecting the settings button on electronic device 1, the terminal device displays user interface 1210, see [link to relevant documentation]. Figure 12C As shown, the user interface 1210 includes selection modes, device modes, and action selections for the electronic device 1 when using the custom wake word. Users can select according to their needs.

[0294] In one possible implementation, the terminal device may also provide the user with a user interface for unified management of wake words in the system. See, as an example, [link to example]. Figure 13The image shows a possible user interface 1310 for unified management of wake words. Users can modify the electronic devices to which the wake words are compatible, the usage mode of the wake words, and the device mode of the wake words as needed. For example, a voice control system may include wake words such as... Figure 13 As shown. The user interface 1310 includes selection controls for setting each wake word, such as... Figure 13 Examples include 1311-1314. In response to user settings actions related to the wake word "Xiao A Xiao A," it displays the current usage mode, device mode, or compatible electronic device for "Xiao A Xiao A," for example, see [link to relevant documentation]. Figure 13 As shown, the current usage mode of "Xiao A Xiao A" is private mode, the device mode is near-field mode, and the compatible electronic devices are electronic device 1 and electronic device 2. Users can change the usage mode, device mode, or compatible electronic devices as needed. It should be noted that when a user changes the usage mode or device mode of the wake word, the server needs to send the updated wake word wake-up threshold to the compatible electronic devices. If the server has configured the wake-up threshold for the wake word for the changed usage mode or device mode, it can send the pre-stored wake-up threshold for that usage mode or device mode to the compatible electronic devices. As an example, when the terminal device responds to the user's operation on the wake word management setting control 723 in the user interface 720, it displays the wake word management user interface 1310.

[0295] In another possible implementation, the terminal device may also provide the user with a user interface for unified management of wake words for electronic devices in the system. As an example, see [link to example]. Figure 14 The image shows a possible user interface 1410 for unified management of electronic devices. Users can modify the wake word used by the electronic device, the action performed when the wake word wakes the electronic device, the usage mode of the wake word, the device mode of the wake word, and other electronic devices compatible with the wake word, according to their needs. For example, the voice control system includes electronic devices such as... Figure 14As shown. The user interface 1410 includes selection controls for setting various electronic devices. In response to a user's setting operation on electronic device 1, the user interface 1410 displays the wake-up word configured for electronic device 1, the wake-up word currently used by the electronic device, and setting controls for wake-up actions and usage scenario information. The user can select the wake-up word and the corresponding action, as well as usage scenario information, according to their needs. The user can change the usage mode, device mode, or compatible electronic device according to their needs. It should be noted that when the user changes the usage mode or device mode of the wake-up word, the server needs to send the updated wake-up threshold and probability compensation value of the modeling unit included in the wake-up word to the compatible electronic device. If the server has configured the wake-up threshold and probability compensation value of the modeling unit included in the wake-up word for the changed usage mode or device mode, it can send the pre-stored wake-up threshold and probability compensation value of the modeling unit included in the wake-up word under the usage mode or device mode to the compatible electronic device. As an example, when the terminal device responds to the user's operation on the wake-up device management setting control 724 in the user interface 720, it displays the wake-up device management user interface 1410.

[0296] In a second possible application scenario, some embodiments allow users to customize the location, local area network, or time period used for the wake word, according to their needs. For example, different locations can correspond to different usage modes. Similarly, different local area networks can correspond to different usage modes. And different time periods can correspond to different usage modes.

[0297] For example, the location and usage scenario can be configured for the same or different wake words. For instance, users can use public or far-field mode at home, and private or near-field mode outside of home.

[0298] For example, configurations can be made for the local area network used with the same or different wake words, as well as the usage scenarios within that local area network. For instance, when electronic devices are connected to a home LAN, they can be configured in public mode or far-field mode. When connected to a company LAN, they can be configured in private mode or near-field mode.

[0299] For example, you can configure the time period and usage scenario for the same or different wake words. For instance, you can configure near-field mode at night and remote mode during the day.

[0300] For example, you can configure the time period and local area network used for the same or different wake words. For instance, when connecting to your home LAN at night, you can configure far-field mode and / or public mode, and in other cases, you can configure near-field mode and / or private mode.

[0301] As an example, see Figure 8F The diagram shows one possible configuration location. Figure 8E This is merely an example and does not impose specific limitations on the selection method or interface display format. For instance, an exclusive selection method could be used, excluding usage scenarios for configurations in other locations besides a particular one. Alternatively, an inclusive selection method could be used, selecting usage scenarios for configurations corresponding to a specific location.

[0302] Furthermore, when the server receives the usage scenario information corresponding to the custom wake word and the user-input location area information, time period information, or local area network information, it can configure the electronic device that the custom wake word applies to. The electronic device can associate and store the usage scenario information, location area information, time period information, local area network information, custom wake word, and wake-up threshold.

[0303] The following details the process by which an electronic device performs a wake-up operation based on a wake-up threshold (and the probability compensation value of the modeling unit included in the wake-up word). See also Figure 15 As shown, taking the first electronic device as an example, the wake word currently configured for the first electronic device is the first wake word.

[0304] 1501, The first electronic device is monitoring voice information.

[0305] In some embodiments, voice information can be monitored using a microphone or microphone array.

[0306] 1502, when the first electronic device determines that the probability of the voice information including the first wake-up word is greater than the first wake-up threshold, it performs the operation of waking up the first electronic device corresponding to the first wake-up word in the first usage scenario; the first wake-up threshold is the wake-up threshold of the first wake-up word in the first usage scenario. For example, the first usage scenario may include a private mode or a public mode, and the first usage scenario may also include a near-field mode or a far-field mode.

[0307] Different wake words have different wake-up thresholds in different usage scenarios. The same wake word may have different wake-up thresholds in different usage scenarios.

[0308] For example, when the probability that the voice information contains a second wake-up word is greater than a second wake-up threshold, the first electronic device performs a second operation to wake up the first electronic device corresponding to the second wake-up word in the second usage scenario; the second wake-up word is a wake-up word configured in the first electronic device in the second usage scenario, and the second wake-up threshold is the wake-up threshold of the second wake-up word in the second usage scenario. The first wake-up word and the second wake-up word can be different or the same.

[0309] In some embodiments, the wake-up threshold varies for the same wake word in different usage scenarios. For example, a single wake word for an electronic device can be configured for different usage scenarios, with different wake-up thresholds for each scenario, thus changing the wake-up effect according to the usage scenario. Users can configure the usage scenarios for using the wake word according to their needs. Again, for example, a single wake word for an electronic device can be configured for different usage scenarios, with different wake-up thresholds for each scenario, thus changing the wake-up effect according to the usage scenario. Users can choose from various usage scenarios for the wake word, such as different location areas, different time periods, or different local area networks.

[0310] In other embodiments, for a single electronic device, different wake words correspond to different usage scenarios, allowing the electronic device to determine the usage scenario based on the wake word. Optionally, different wake words may also correspond to different location areas, different time periods, or different local area networks.

[0311] In some embodiments, the first wake-up word and the first wake-up threshold of the first electronic device can be determined by itself, for example, the first electronic device is a terminal device. The specific determination method is as described above and will not be repeated here.

[0312] In other embodiments, the first wake-up word of the first electronic device may also be determined by the server and sent to the first electronic device. The first electronic device may be connected to the server. For example, the server may be a cloud server, and the first electronic device may register with the cloud server.

[0313] In other embodiments, the first wake-up word of the first electronic device may also be determined by the terminal device and sent to the first electronic device. The first electronic device may establish a connection with the terminal device, such as a short-range communication connection, for example, Bluetooth, WIFI, etc.

[0314] In one possible implementation, the first electronic device can determine that the probability of the voice information including the first wake-up word is greater than the first wake-up threshold by means of the following:

[0315] Each wake-up word configured on the first electronic device and its corresponding usage scenario information are determined, wherein each wake-up word includes the first wake-up word. Further, wake-up thresholds for each usage scenario corresponding to each wake-up word are obtained. The usage scenario information for each wake-up word corresponds to a different wake-up threshold.

[0316] As an example, a wake word is configured for the first electronic device, which corresponds to only one use case.

[0317] As another example, the wake word configured on the first electronic device is related to the currently registered user of the first electronic device. As the registered user changes, the configured wake word or the usage scenario of the configured wake word changes on the first electronic device. The usage scenarios of the same wake word for different users registered on the first electronic device can be different or the same. Furthermore, when determining each wake word configured on the first electronic device and the usage scenario information corresponding to each wake word, the wake words corresponding to the first user on the first electronic device and the usage scenario information corresponding to each wake word can be determined based on the user information of the first user using the first electronic device.

[0318] In one possible implementation, when the first electronic device determines that the probability of the voice information including the first wake-up word is greater than a first wake-up threshold, it can determine this by using a probability compensation value set. The probability compensation value set includes probability compensation values ​​for N1 modeling units included in the first wake-up word in the first usage scenario, where N1 is a positive integer. The probability compensation value for the first modeling unit is used to compensate for the acoustic posterior probability of the first modeling unit when determining that the voice information includes the first modeling unit. The first modeling unit is any one of the N1 modeling units.

[0319] For example, the first wake-up word includes xiao, hua, xiao, hua; the modeling units in the first wake-up word include xiaohuaxiaohua; taking the first modeling unit as x as an example, after the first electronic device determines the acoustic posterior probability of x through the acoustic probability model, it then uses the probability compensation value of x to compensate for the acoustic posterior probability of x determined by the acoustic probability model, which is used as the acoustic posterior probability of the modeling unit x. Then, it further determines whether the probability of receiving voice information containing the custom wake-up word reaches the wake-up threshold corresponding to the custom wake-up word by using the determined acoustic posterior probabilities of each modeling unit, thereby deciding whether to wake up the electronic device.

[0320] The following is combined Figure 4BThe application framework layer provides a detailed explanation of the voice wake-up process.

[0321] A1, receives the voice information detected by the microphone through the frame layer object;

[0322] The framework layer object can be the voice wake-up engine in the application framework layer, or the voice wake-up engine in HMS core.

[0323] As an example, the voice wake-up engine can also be located at the application layer.

[0324] A2, through the framework layer object, determine at least one wake word configured on the first electronic device and the usage scenario information corresponding to the at least one wake word, wherein the at least one wake word includes the first wake word.

[0325] A3 retrieves the wake-up threshold corresponding to the usage scenario information of each wake-up word stored in the framework layer object.

[0326] The framework layer object can obtain one or more wake words configured on the first electronic device from the storage service, as well as the wake-up threshold under the usage scenario information corresponding to each wake word.

[0327] As an example, a wake word is configured for the first electronic device, the use case of which has been determined.

[0328] As another form of distance, the usage scenarios of the same wake word by different users on the first electronic device can be different or the same.

[0329] A4, by determining through the framework layer object that the probability of the voice information including the first wake-up word is greater than the first wake-up threshold and the probability of the voice information including other wake-up words is less than or equal to the wake-up threshold of the other wake-up words.

[0330] It is understood that, in order to achieve the functions in the above method embodiments, the electronic device includes hardware structures and / or software modules corresponding to perform each function. Those skilled in the art should readily recognize that, based on the modules and method steps described in conjunction with the embodiments disclosed in this application, this application can be implemented in hardware or a combination of hardware and computer software. Whether a function is executed by hardware or by computer software driving hardware depends on the specific application scenario and design constraints of the technical solution.

[0331] Based on the same inventive concept as the methods described above, such as Figure 16 As shown, this application embodiment also provides a voice control device 1600. The voice control device is applied in electronic devices or terminal devices with touch display functions, such as... Figure 2In the electronic device 100 shown, or Figure 3 The terminal device 200 and apparatus 1600 shown can be used to implement the functions of the terminal device or electronic device in the above method embodiments, and thus can achieve the beneficial effects of the above method embodiments. For example, the voice control device may include a listening module 1601 and a processing module 1602.

[0332] In some embodiments:

[0333] The 1601 monitoring module is used to monitor voice information.

[0334] Processing module 1602 is used to determine that when the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold, to perform a first operation to wake up the first electronic device corresponding to the first wake-up word in the first usage scenario; the first wake-up word is a wake-up word configured in the first electronic device in the first usage scenario, and the first wake-up threshold is the wake-up threshold of the first wake-up word in the first usage scenario;

[0335] Processing module 1602 is configured to perform a second operation to wake up the first electronic device corresponding to the second wake-up word in a second usage scenario when it is determined that the probability of the voice information containing the second wake-up word is greater than the second wake-up threshold; the second wake-up word is a wake-up word configured in the first electronic device in the second usage scenario, and the second wake-up threshold is the wake-up threshold of the second wake-up word in the second usage scenario; wherein, the first wake-up threshold is different from the second wake-up threshold.

[0336] In one possible implementation, the first wake word is different from the second wake word.

[0337] In one possible implementation, when the first electronic device is connected to the first local area network, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is connected to the second local area network, the usage scenario of the first electronic device is the second usage scenario; or...

[0338] When the first electronic device is located in the first location area, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is located in the second location area, the usage scenario of the first electronic device is the second usage scenario.

[0339] In one possible implementation, before determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold, the processing module 1602 is further configured to acquire information input by the user, the information including wake-up word data of the first wake-up word and usage scenario information of the first wake-up word, wherein the usage scenario information indicates the first usage scenario; the wake-up word data and the usage scenario information are used to determine the first wake-up threshold.

[0340] In one possible implementation, a transceiver module 1603 may be included, which is used to send information input by the user, including the wake word data and the usage scenario information, to the server; and to receive the first wake-up threshold determined by the server based on the wake word data and the usage scenario information.

[0341] In one possible implementation, the processing module 1602 is further configured to: perform synthetic processing on the wake-up word data according to the usage scenario information to obtain a synthetic speech dataset; and determine a first wake-up threshold for the first wake-up word in a first usage scenario based on the synthetic speech dataset.

[0342] In one possible implementation, the usage scenario information includes usage mode and / or device mode; wherein, the usage mode indicates a private mode or a public mode, the private mode is used to indicate that the first wake word is adapted to a private user, and the public mode is used to indicate that the first wake word is adapted to multiple users; the device mode indicates a proximity mode or a far-field mode, the wake-up distance of the proximity mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

[0343] In one possible implementation, the usage scenario information includes the usage mode, the usage mode indicating a public mode, and the processing module 1602 is specifically used for: if the wake-up word data is in text format, performing text-to-speech (TTS) processing on the wake-up word data to obtain first speech data, performing data augmentation processing on the first speech data to obtain multiple first speech synthesis data, the synthesis dataset including the multiple first speech synthesis data and the first speech data; or, if the wake-up word data is in speech format, performing data augmentation processing on the wake-up word data to obtain multiple second speech synthesis data; the synthesis dataset including the multiple second speech synthesis data and the wake-up word data.

[0344] In one possible implementation, the usage scenario information includes the usage mode, the usage mode indicating a private mode, and the wake word data is in speech format; the processing module 1602 is specifically used to: perform sound cloning processing on the wake word data to obtain multiple third speech synthesis data; the synthesis dataset includes the multiple third speech synthesis data and the wake word data.

[0345] In one possible implementation, the usage scenario information includes a usage mode and a device mode, wherein the usage mode indicates a public mode and the device mode indicates a far-field speaking mode; the processing module 1602 is specifically configured to: when the wake-up word data is in text format, perform TTS processing on the wake-up word data to obtain second speech data, perform data augmentation processing on the second speech data to obtain multiple fourth speech synthesis data, and convert the fourth speech synthesis data and the second speech data into far-field speaking speech to obtain the synthesized speech dataset; or, when the wake-up word data is in speech format, perform data augmentation processing on the wake-up word data to obtain multiple fifth speech synthesis data, and convert the fourth speech synthesis data and the wake-up word data into far-field speaking speech to obtain the synthesized speech dataset.

[0346] In one possible implementation, a display module 1604 is further included for displaying a first user interface, the first user interface including an input control for a user to input a first wake-up word and a first selection control for selecting a usage scenario of the first wake-up word; the processing module 1602 is further used to obtain the wake-up word data and the usage scenario information of the first wake-up word in response to the user's operation of inputting the first wake-up word through the input control and selecting the first usage scenario through the first selection control.

[0347] In one possible implementation, the display module 1604 is further configured to display a control main interface, the control main interface including a first option for adding a custom wake word; the processing module 1602 is further configured to display the first user interface in response to the user selecting the first option.

[0348] In one possible implementation, the main control interface further includes a second option for selecting N wake words defined for the first electronic device; the processing module 1602 is further configured to respond to the operation of selecting the second option, and the display module 1604 is further configured to display a third user interface; wherein the third user interface includes N wake word options, and different wake word options are used to select different wake words; the third user interface further includes a second selection control for selecting the usage scenario corresponding to each wake word.

[0349] In one possible implementation, the processing module 1602 is specifically configured to determine, based on a probability compensation value set, that the probability of the voice information including the first wake-up word is greater than the first wake-up threshold; wherein, the probability compensation value set includes probability compensation values ​​of N1 modeling units used in the first usage scenario to determine whether the voice information includes the first wake-up word, where N1 is a positive integer; the probability compensation value of the first modeling unit is used to compensate the acoustic posterior probability of the first modeling unit when determining the acoustic posterior probability of the voice information including the first modeling unit, the acoustic posterior probability of the first modeling unit is used to determine the probability of the first wake-up word, and the first modeling unit is any one of the N1 modeling units.

[0350] In one possible implementation, the processing module 1602 is further configured to determine a set of probability compensation values ​​for the first wake-up word in a first usage scenario based on the synthesized speech dataset.

[0351] In one possible implementation, the processing module 1602 is specifically configured to: extract speech feature information of each basic computing unit in the K synthesized speech data; determine the acoustic posterior probability of the modeling unit corresponding to the M basic computing units based on the speech feature data of each basic computing unit; the modeling unit corresponding to the M basic computing units is the modeling unit included in the first wake-up word; obtain the average value of the acoustic posterior probability of the j-th modeling unit corresponding to the M basic computing units; the i-th modeling unit is the j-th of the N1 modeling units, where j is a positive integer less than or equal to N1; obtain the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units; and obtain the probability compensation value of the j-th modeling unit based on the average value of the acoustic posterior probability of the j-th modeling unit and the maximum value.

[0352] In one possible implementation, the processing module 1602 is specifically configured to perform probability compensation on the acoustic posterior probabilities of the M basic computing units corresponding to the j-th modeling unit according to the probability compensation value of the j-th modeling unit to obtain the compensated acoustic posterior probability of the j-th modeling unit, where j takes all positive integers less than or equal to N1; determine the probability that the synthesized speech dataset includes the first wake-up word according to the compensated acoustic posterior probabilities of the N1 modeling units, and the probability that the synthesized speech dataset includes the first wake-up word is used to determine the first wake-up threshold.

[0353] In one possible implementation, the transceiver module 1603 is further configured to receive a control message sent by the server before determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold. The control message includes the first wake-up word, usage scenario information for indicating a first usage scenario of the first wake-up word, and a first wake-up threshold of the first wake-up word in the first usage scenario.

[0354] In one possible implementation, the processing module 1602 is specifically configured to receive the voice information detected by the microphone through a frame layer object; determine, through the frame layer object, at least one wake-up word configured on the first electronic device and usage scenario information corresponding to the at least one wake-up word, wherein the at least one wake-up word includes the first wake-up word; obtain, through the frame layer object, a wake-up threshold corresponding to the usage scenario information of each stored wake-up word; and determine, through the frame layer object, that the probability that the voice information includes the first wake-up word is greater than the first wake-up threshold and the probability that the voice information includes other wake-up words is less than or equal to the wake-up threshold of the other wake-up words.

[0355] Based on this, this application also provides a voice control device, see [link to relevant documentation]. Figure 17 As shown, the voice control device 1700 includes a processor 1701, a microphone 1702, and a memory 1703. The memory 1703 stores instructions or programs executed by the processor 1701, or input data required for the processor 1701 to run instructions or programs, or data generated after the processor 1701 runs instructions or programs. The microphone is used to listen to voice information. The processor 1701 runs the instructions or programs stored in the memory 1703 to perform the functions of the electronic device or terminal device in any of the above method embodiments. The voice control device 1700 may also include a transceiver 1704 for sending and receiving data, such as sending user-inputted information, or receiving wake-up words and wake-up thresholds. The transceiver 1704 can use any device with transceiver functions for communicating with other devices or communication networks, such as Ethernet, radio access network (RAN), wireless local area network (WLAN), wired access network, etc. When the voice control device 1700 is used to implement the functions of the terminal device, the voice control device 1700 may also include a display screen 1705, which may have a touch detection function.

[0356] In one example, processor 1701 performs the functions of the aforementioned listening module 1601, processing module 1602, transceiver module 1603, and display module 1604. In another example, the function of listening module 1601 is implemented by microphone 1702, the function of processing module 1602 is implemented by processor 1701, the function of transceiver module 1603 is implemented by transceiver 1704, and the function of display module 1604 can be implemented by display screen 1705.

[0357] It is understood that the processor in the embodiments of this application can be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, transistor logic devices, hardware components, or any combination thereof. A general-purpose processor can be a microprocessor or any conventional processor.

[0358] The method steps in the embodiments of this application can be implemented in hardware or by a processor executing software instructions. The software instructions can consist of corresponding software modules, which can be stored in random access memory (RAM), flash memory, read-only memory (ROM), programmable read-only memory (PROM), erasable programmable read-only memory (EPROM), electrically erasable programmable read-only memory (eEPROM), registers, hard disks, portable hard disks, CD-ROMs, or any other form of storage medium known in the art. An exemplary storage medium is coupled to a processor, enabling the processor to read information from and write information to the storage medium. Of course, the storage medium can also be a component of the processor. The processor and storage medium can reside in an ASIC. Additionally, the ASIC can reside in a terminal device. Alternatively, the processor and storage medium can exist as discrete components in the terminal device.

[0359] In the above embodiments, implementation can be achieved entirely or partially through software, hardware, firmware, or any combination thereof. When implemented using software, it can be implemented entirely or partially in the form of a computer program product. The computer program product includes one or more computer programs or instructions. When the computer program or instructions are loaded and executed on a computer, the processes or functions described in the embodiments of this application are performed entirely or partially. The computer can be a general-purpose computer, a special-purpose computer, a computer network, a user equipment, or other programmable device. The computer program or instructions can be stored in a computer-readable storage medium or transferred from one computer-readable storage medium to another. For example, the computer program or instructions can be transferred from one website, computer, server, or data center to another website, computer, server, or data center via wired or wireless means. The computer-readable storage medium can be any available medium that a computer can access or a data storage device such as a server or data center that integrates one or more available media. The available medium can be a magnetic medium, such as a floppy disk, hard disk, or magnetic tape; it can also be an optical medium, such as a digital video disc (DVD); or it can be a semiconductor medium, such as a solid-state drive (SSD).

[0360] In the various embodiments of this application, unless otherwise specified or logically conflicting, the terminology and / or descriptions between different embodiments are consistent and can be referenced mutually. Technical features in different embodiments can be combined to form new embodiments based on their inherent logical relationships. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion, such as including a series of steps or units. A method, system, product, or device is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to these processes, methods, products, or devices.

[0361] Although this application has been described in conjunction with specific features and embodiments, it is obvious that various modifications and combinations can be made therein without departing from the spirit and scope of this application. Accordingly, this specification and drawings are merely illustrative examples of the solutions defined by the appended claims and are to be considered as covering any and all modifications, variations, combinations, or equivalents within the scope of this application.

[0362] Obviously, those skilled in the art can make various modifications and variations to this application without departing from the scope of this application. Therefore, if these modifications and variations of the embodiments of this application fall within the scope of the claims of this application and their equivalents, this application also intends to include these modifications and variations.

Claims

1. A voice control method, characterized in that, include: Listen to voice information; When the probability that the voice information contains a first wake-up word is greater than a first wake-up threshold, a first operation is performed to wake up the first electronic device corresponding to the first wake-up word in the first usage scenario; the first wake-up word is a wake-up word configured in the first electronic device in the first usage scenario, and the first wake-up threshold is the wake-up threshold of the first wake-up word in the first usage scenario; When the probability that the voice information contains a second wake-up word is greater than a second wake-up threshold, a second operation is performed to wake up the first electronic device corresponding to the second wake-up word in the second usage scenario; the second wake-up word is a wake-up word configured in the first electronic device in the second usage scenario, and the second wake-up threshold is the wake-up threshold of the second wake-up word in the second usage scenario; The first wake-up threshold is different from the second wake-up threshold; Before determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold, the method further includes: The system acquires user-inputted information, including wake-up word data and usage scenario information for the first wake-up word, wherein the usage scenario information indicates the first usage scenario; the wake-up word data and the usage scenario information are used to determine the first wake-up threshold; the usage scenario information includes usage mode and / or device mode; wherein the usage mode indicates a private mode or a public mode, the private mode indicates that the first wake-up word is adapted to one private user, and the public mode indicates that the first wake-up word is adapted to multiple users; the device mode indicates a near-field mode or a far-field mode, wherein the wake-up distance of the near-field mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

2. The method as described in claim 1, characterized in that, The first wake word is different from the second wake word.

3. The method as described in claim 1 or 2, characterized in that, When the first electronic device is connected to the first local area network, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is connected to the second local area network, the usage scenario of the first electronic device is the second usage scenario; or... When the first electronic device is located in the first location area, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is located in the second location area, the usage scenario of the first electronic device is the second usage scenario.

4. The method as described in claim 1, characterized in that, The method further includes: Send the user input information, including the wake word data and the usage scenario information, to the server; Receive the first wake-up threshold determined by the server based on the wake-up word data and the usage scenario information.

5. The method as described in claim 1, characterized in that, The method further includes: Based on the usage scenario information, the wake word data is synthesized to obtain a synthetic speech dataset; The first wake-up threshold for the first wake-up word in the first usage scenario is determined based on the synthesized speech dataset.

6. The method as described in claim 1, characterized in that, The usage scenario information includes the usage mode, which indicates a public mode. Based on the usage scenario information, the wake word data is synthesized to obtain a synthesized speech dataset, including: If the wake-up word data is in text format, text-to-speech (TTS) processing is performed on the wake-up word data to obtain first speech data. Data augmentation processing is then performed on the first speech data to obtain multiple first speech synthesis data sets. The synthesis dataset includes the multiple first speech synthesis data sets and the first speech data set; or... If the wake word data is in speech format, data augmentation processing is performed on the wake word data to obtain multiple second speech synthesis data; the synthesis dataset includes the multiple second speech synthesis data and the wake word data.

7. The method as described in claim 1, characterized in that, The usage scenario information includes the usage mode, the usage mode indicating a private mode, and the wake word data in speech format; the wake word data is synthesized based on the usage scenario information to obtain a synthesized speech dataset, including: The wake word data is processed by sound cloning to obtain multiple third-party speech synthesis data; The synthetic dataset includes the plurality of third-party speech synthesis data and the wake word data.

8. The method as described in claim 1, characterized in that, The usage scenario information includes usage mode and device mode, wherein the usage mode indicates public mode and the device mode indicates far-reaching mode; Based on the usage scenario information, the wake word data is synthesized to obtain a synthesized speech dataset, including: When the wake word data is in text format, TTS processing is performed on the wake word data to obtain second speech data, data augmentation processing is performed on the second speech data to obtain multiple fourth speech synthesis data, and the fourth speech synthesis data and the second speech data are converted into far-field speech to obtain the synthesized speech dataset. or, When the wake-up word data is in speech format, data augmentation processing is performed on the wake-up word data to obtain multiple fifth speech synthesis data, and the fourth speech synthesis data and the wake-up word data are converted into far-field speech to obtain the synthesized speech dataset.

9. The method according to any one of claims 1-8, characterized in that, The process of obtaining user input includes: Display a first user interface, the first user interface including an input control for a user to input a first wake word and a first selection control for selecting the usage scenario of the first wake word; In response to the user's input of a first wake-up word via an input control and selection of a first usage scenario via the first selection control, the wake-up word data and the usage scenario information of the first wake-up word are obtained.

10. The method as described in claim 9, characterized in that, The method further includes: The main control interface is displayed, which includes a first option for adding a custom wake word; In response to the user selecting the first option, the first user interface is displayed.

11. The method as described in claim 10, characterized in that, The main control interface also includes a second option for selecting and managing the N wake words defined by the first electronic device; In response to the action of selecting the second option, a third user interface is displayed; The third user interface includes N wake word options, with different wake word options used to select different wake words; The third user interface also includes a second selection control for selecting the usage scenario corresponding to each wake word.

12. The method according to any one of claims 1-8, characterized in that, The determination that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold includes: The probability that the voice information includes the first wake-up word is greater than the first wake-up threshold, based on the probability compensation value set. The probability compensation value set includes probability compensation values ​​for N1 modeling units used in the first usage scenario to determine whether the voice information includes the first wake-up word, where N1 is a positive integer; The probability compensation value of the first modeling unit is used to compensate the acoustic posterior probability of the first modeling unit when determining that the acoustic posterior probability of the first modeling unit is included in the speech information. The acoustic posterior probability of the first modeling unit is used to determine the probability of the first wake-up word. The first modeling unit is any one of the N1 modeling units.

13. The method as described in claim 12, characterized in that, The method further includes: Determine the set of probability compensation values ​​for the first wake-up word in the first usage scenario based on the synthesized speech dataset; The synthesized speech dataset is obtained by synthesizing wake word data of the first wake word input by the user based on usage scenario information used to indicate the first usage scenario.

14. The method as described in claim 13, characterized in that, The step of determining the probability compensation value set of the first wake-up word in the first usage scenario based on the synthesized speech dataset, wherein the synthesized speech dataset includes K synthesized speech information, including: Extract the speech feature information of each basic computing unit from the K synthesized speech data; The acoustic posterior probability of the modeling unit corresponding to the M basic computing units is determined based on the speech feature data of each basic computing unit; the modeling unit corresponding to the M basic computing units is the modeling unit included in the first wake-up word; Obtain the average acoustic posterior probability of the M basic computational units corresponding to the j-th modeling unit; the j-th modeling unit is the j-th of the N1 modeling units, and j takes all positive integers less than or equal to N1; Obtain the maximum value among the average values ​​of the acoustic posterior probabilities of the N1 modeling units; The probability compensation value of the j-th modeling unit is obtained by the average value and the maximum value of the acoustic posterior probability of the j-th modeling unit.

15. The method as described in claim 14, characterized in that, Determining the first wake-up threshold for the first wake-up word in the first usage scenario based on the synthesized speech dataset includes: Based on the probability compensation value of the j-th modeling unit, the acoustic posterior probability of the j-th modeling unit corresponding to the M basic computing units is probabilistically compensated to obtain the compensated acoustic posterior probability of the j-th modeling unit, where j takes all positive integers less than or equal to N1. The probability that the synthesized speech dataset includes the first wake-up word is determined based on the compensated acoustic posterior probability of N1 modeling units, and the probability that the synthesized speech dataset includes the first wake-up word is used to determine the first wake-up threshold.

16. The method according to any one of claims 1-3, characterized in that, Before determining that the probability of the voice information containing the first wake-up word is greater than the first wake-up threshold, the method further includes: The system receives a control message sent by the server. The control message includes the first wake-up word, usage scenario information for indicating a first usage scenario of the first wake-up word, and a first wake-up threshold for the first wake-up word in the first usage scenario.

17. The method according to any one of claims 1-8, characterized in that, Determining that the probability of the voice information containing the first wake-up word is greater than a first wake-up threshold includes: The voice information detected by the microphone is received through the frame layer object; The framework layer object determines at least one wake word configured on the first electronic device and the usage scenario information corresponding to the at least one wake word, wherein the at least one wake word includes the first wake word; The wake-up threshold corresponding to the usage scenario information of each wake-up word stored in the framework layer object is obtained. The probability that the voice information includes the first wake-up word is greater than the first wake-up threshold, and the probability that the voice information includes other wake-up words is less than or equal to the wake-up threshold of the other wake-up words, is determined by the framework layer object.

18. A voice control device, characterized in that, Including memory and processor; The memory is used to store programs or instructions; The processor is configured to invoke the program or instructions to cause the voice control device to perform the method as described in any one of claims 1 to 17.

19. A voice control method, characterized in that, include: Configure a first wake-up word and a first wake-up threshold corresponding to the first wake-up word to a first electronic device, wherein the first wake-up threshold is the wake-up threshold of the first wake-up word in a first usage scenario; Configure a second wake-up word and a second wake-up threshold corresponding to the second wake-up word to the first electronic device; the second wake-up threshold is the wake-up threshold of the second wake-up word in a second usage scenario; wherein, the first wake-up threshold is different from the second wake-up threshold; The method further includes: acquiring information input by the user, the information including wake word data of the first wake word and usage scenario information of the first wake word, wherein the usage scenario information indicates the first usage scenario; and determining the first wake-up threshold based on the usage scenario information and the wake word data. The usage scenario information includes usage mode and / or device mode; wherein, the usage mode indicates a private mode or a public mode, the private mode is used to indicate that the first wake word is adapted to a private user, and the public mode is used to indicate that the first wake word is adapted to multiple users; the device mode indicates a near-field mode or a far-field mode, the wake-up distance of the near-field mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

20. A voice control device, characterized in that, Including memory and processor; The memory is used to store programs or instructions; The processor is configured to invoke the program or instructions to cause the voice control device to perform the method as described in claim 19.

21. A voice control system, characterized in that, It includes a management device and a first electronic device, wherein the management device establishes a communication connection with the first electronic device; A management device is configured to configure a first wake-up word and a first wake-up threshold corresponding to the first wake-up word to the first electronic device, wherein the first wake-up threshold is the wake-up threshold of the first wake-up word in a first usage scenario; The first electronic device is configured to perform a first operation to wake up the first electronic device in a first usage scenario when the probability of detecting the first voice information including the first wake-up word is greater than a first wake-up threshold. The management device is also configured to configure a second wake-up word and a second wake-up threshold corresponding to the second wake-up word to the first electronic device; the second wake-up threshold is the wake-up threshold of the second wake-up word in a second usage scenario; The first electronic device is further configured to perform a second operation to wake up the first electronic device in the second usage scenario when the probability of detecting the second voice information including the second wake-up word is greater than the second wake-up threshold. The first wake-up threshold is different from the second wake-up threshold; The management device is a server, and the system also includes a fourth electronic device; The fourth electronic device is used to acquire information input by the user, the information including wake word data of the first wake word and usage scenario information of the first wake word, wherein the usage scenario information indicates the first usage scenario; and is used to send the information input by the user to the server. The server is configured to determine the first wake-up threshold based on the wake-up word data and the usage scenario information, and send the first wake-up threshold to the first electronic device; The usage scenario information includes usage mode and / or device mode; wherein, the usage mode indicates a private mode or a public mode, the private mode is used to indicate that the first wake word is adapted to a private user, and the public mode is used to indicate that the first wake word is adapted to multiple users; the device mode indicates a near-field mode or a far-field mode, the wake-up distance of the near-field mode is less than the wake-up distance of the far-field mode; the wake-up distance is the distance between the sound source and the first electronic device.

22. The system as claimed in claim 21, characterized in that, The first wake word is different from the second wake word.

23. The system as claimed in claim 21, characterized in that, When the first electronic device is connected to the first local area network, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is connected to the second local area network, the usage scenario of the first electronic device is the second usage scenario; or... When the first electronic device is located in the first location area, the usage scenario of the first electronic device is the first usage scenario; when the first electronic device is located in the second location area, the usage scenario of the first electronic device is the second usage scenario.

24. The system according to any one of claims 21-23, characterized in that, The management device is a second electronic device, which is further used for: Before determining that the probability of the first wake-up word being contained in the first voice information is greater than the first wake-up threshold, information input by the user is obtained. The information includes wake-up word data of the first wake-up word and usage scenario information of the first wake-up word. The usage scenario information indicates the first usage scenario. The first wake-up threshold is determined based on the wake-up word data and the usage scenario information, and then the first wake-up threshold is sent to the first electronic device.

25. The system according to any one of claims 21-23, characterized in that, The management device is a server, and the system also includes a third electronic device; The management device is further configured to configure a third wake-up word and a third wake-up threshold corresponding to the third wake-up word to the third electronic device, wherein the third wake-up threshold is the wake-up threshold of the third wake-up word in a third usage scenario; The third electronic device is also used to perform a third operation to wake up the third electronic device in the third usage scenario when the probability of listening to the third voice information including the third wake-up word is greater than the third wake-up threshold. The third wake-up threshold is different from the first wake-up threshold and the second wake-up threshold.

26. The system as claimed in claim 21, characterized in that, The information input by the user also includes information about the electronic device to which the first wake word is applied, and the electronic device information indicates the first electronic device.

27. The system as claimed in claim 26, characterized in that, The system also includes a fifth electronic device; the electronic device information further indicates the fifth electronic device; The server is also configured to send the first wake word and the first wake threshold to the fifth electronic device.

28. The system as claimed in claim 26, characterized in that, The server is specifically used for: Based on the usage scenario information, the wake word data is synthesized to obtain a synthetic speech dataset; The first wake-up threshold for the first wake-up word in the first usage scenario is determined based on the synthesized speech dataset.

29. The system as claimed in claim 21, characterized in that, The usage scenario information includes the usage mode, the usage mode indicating a public mode, and the server is specifically used for: If the wake word data is in text format, the wake word data is processed by text-to-speech (TTS) to obtain first speech data. The first speech data is then processed by data augmentation to obtain multiple first speech synthesis data. The synthesis dataset includes the multiple first speech synthesis data and the first speech data. or, If the wake word data is in speech format, data augmentation processing is performed on the wake word data to obtain multiple second speech synthesis data; the synthesis dataset includes the multiple second speech synthesis data and the wake word data.

30. The system as claimed in claim 21, characterized in that, The usage scenario information includes the usage mode, the usage mode indicating a private mode, and the wake-up word data in voice format; the server is specifically used for: The wake word data is processed by sound cloning to obtain multiple third-party speech synthesis data; The synthetic dataset includes the plurality of third-party speech synthesis data and the wake word data.

31. The system as described in claim 21, characterized in that, The usage scenario information includes usage mode and device mode, wherein the usage mode indicates public mode and the device mode indicates far-reaching mode; The server is specifically used for: When the wake word data is in text format, TTS processing is performed on the wake word data to obtain second speech data, data augmentation processing is performed on the second speech data to obtain multiple fourth speech synthesis data, and the fourth speech synthesis data and the second speech data are converted into far-field speech to obtain the synthesized speech dataset. or, When the wake-up word data is in speech format, data augmentation processing is performed on the wake-up word data to obtain multiple fifth speech synthesis data, and the fourth speech synthesis data and the wake-up word data are converted into far-field speech to obtain the synthesized speech dataset.

32. The system according to any one of claims 26-31, characterized in that, The fourth electronic device is specifically used for: Display a first user interface, the first user interface including an input control for a user to input a first wake word and a first selection control for selecting the usage scenario of the first wake word; In response to the user's input of a first wake-up word via the input control and selection of a first usage scenario via the first selection control, the wake-up word data and the usage scenario information of the first wake-up word are obtained.

33. The system as described in claim 32, characterized in that, The management device is a server, and the fourth electronic device is also used for: Display a second user interface, which includes a first option for adding a custom wake word; In response to the user selecting the first option, the first user interface is displayed.

34. The system according to any one of claims 21-23, characterized in that, The management device is a server, and the system also includes a fourth electronic device; The fourth electronic device is configured to display a third user interface, the third user interface including a second option for selecting N wake words defined by the first electronic device for management; the fourth user interface is displayed in response to the operation of selecting the second option; The fourth user interface includes N wake word options, with different wake word options used to select different wake words, and the N wake word options include options for the first wake word; The fourth electronic device is further configured to send control information to the management device in response to the option of selecting a second wake-up word, the control information being used to instruct the first electronic device to configure a second wake-up word; The management device is specifically used to configure a second wake-up word and a second wake-up threshold corresponding to the second wake-up word to the first electronic device according to the control information.

Images