Voice wake-up method and device, equipment, storage medium and program product

Abstract

The embodiment of the invention provides a voice wake-up method and device, equipment, a storage medium and a program product, and the method comprises the steps: firstly determining a wake-up word set by a user based on an actual voice wake-up demand, then processing the collected multi-channel voice data, inputting the processed multi-channel voice data into a pre-trained model, and determining whether to wake up or not through the model, thereby achieving the precise wake-up, and improving the user experience. The problem that a user feels that voice wake-up is not intelligent due to wake-up caused by false triggering is avoided, and the use experience of the user is improved.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence, and in particular to a voice wake-up method, device, equipment, storage medium, and program product. Background Technology

[0002] Currently, AI assistants installed in terminal devices are a common type of software. For example, the AI ​​assistant that comes with the mobile phone operating system can be set with a wake word. After the wake word is triggered, the AI ​​assistant will respond or perform other next steps.

[0003] However, existing wake-up methods are usually based on no front-end processing or single-channel processing, which can lead to no response or false triggering after a call. This makes users feel that voice wake-up is not flexible enough and cannot meet their actual voice wake-up needs. Summary of the Invention

[0004] This application provides a method, apparatus, device, storage medium, and program product for reminding voice wake-up, in order to solve the above-mentioned technical problems, improve the triggering flexibility of voice wake-up, and meet the actual reminder needs of users.

[0005] In a first aspect, embodiments of this application provide a voice wake-up method, including:

[0006] In response to a voice wake-up operation, multi-channel voice data is acquired, the multi-channel voice data including at least two audio signals, each of which is independently acquired by a corresponding audio acquisition unit; based on the multi-channel voice data, multi-channel feature data is generated, the multi-channel feature data being used to characterize the time-frequency features of at least two audio signals; the multi-channel feature data is processed by a first model to obtain a voice wake-up word, and the corresponding target function is triggered based on the voice wake-up word.

[0007] In one possible implementation, generating multi-channel feature data based on the multi-channel speech data includes: performing blind source separation on the multi-channel speech data to obtain at least two audio components corresponding to the audio signal; obtaining the effective audio components corresponding to each audio signal, wherein the effective audio components correspond to human voice signals; and generating multi-channel feature data based on the effective audio components.

[0008] In one possible implementation, obtaining the effective audio components corresponding to each of the audio signals includes: extracting features from each audio component of the audio signal to obtain audio features corresponding to the audio components; loading a user voice model and determining the effective audio components by calculating the feature similarity between the user voice model and the audio features;

[0009] In one possible implementation, generating multi-channel feature data based on the effective audio components includes: performing bandpass filtering on the effective audio components based on the user voice model to generate bandpass effective audio components; performing frequency domain transformation on the bandpass effective audio components to generate corresponding time-frequency domain matrices; and generating multi-channel feature data based on the time-frequency domain matrices corresponding to each bandpass effective audio component.

[0010] In one possible implementation, generating multi-channel feature data based on the multi-channel speech data includes: performing frequency domain transformation on the audio signals in the multi-channel speech data to generate corresponding time-frequency domain matrices; fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data; and generating multi-channel feature data based on the fused data.

[0011] In one possible implementation, the fused data includes an average time-frequency domain matrix or an extended time-frequency domain matrix. The step of fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data includes: averaging the time-frequency domain matrices corresponding to the at least two audio signals to obtain an average time-frequency domain matrix, wherein the matrix point values ​​in the average time-frequency domain matrix are the average of the matrix point values ​​at corresponding positions in each of the time-frequency domain matrices; or, concatenating the time-frequency domain matrices corresponding to the at least two audio signals to obtain an extended time-frequency domain matrix.

[0012] In one possible implementation, fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data includes: creating a corresponding information item for each matrix point in the time-frequency domain matrix, the information item containing the matrix point value of the time-frequency domain matrix corresponding to at least two audio signals at that position, and reference information corresponding to each audio signal, the reference information representing the audio response curve of the audio acquisition unit that acquires the audio signals; and generating fused data based on the information item corresponding to each matrix point.

[0013] Secondly, embodiments of this application provide a voice wake-up device, including:

[0014] The acquisition module is used to acquire multi-channel voice data in response to a voice wake-up operation. The multi-channel voice data includes at least two audio signals, and each audio signal is independently acquired by a corresponding audio acquisition unit.

[0015] The calculation module generates multi-channel feature data based on the multi-channel speech data, the multi-channel feature data being used to characterize the time-frequency features of at least two audio signals;

[0016] The execution module processes the multi-channel feature data through the first model to obtain the voice wake-up word, and triggers the corresponding target function based on the voice wake-up word.

[0017] In one possible implementation, when the computing module generates multi-channel feature data based on the multi-channel speech data, it is specifically used to: perform blind source separation on the multi-channel speech data to obtain at least two audio components corresponding to the audio signal; obtain the effective audio components corresponding to each audio signal, wherein the effective audio components correspond to human voice signals; and generate multi-channel feature data based on the effective audio components.

[0018] In one possible implementation, the calculation module, in acquiring the effective audio components corresponding to each audio signal, is specifically used for: extracting features from each audio component of the audio signal to obtain audio features corresponding to the audio components; loading a user voice model, and determining the effective audio components by calculating the feature similarity between the user voice model and the audio features;

[0019] In one possible implementation, the computing module generates multi-channel feature data based on the effective audio components, specifically by: performing bandpass filtering on the effective audio components based on the user's voice model to generate bandpass effective audio components; performing frequency domain transformation on the bandpass effective audio components to generate corresponding time-frequency domain matrices; and generating multi-channel feature data based on the time-frequency domain matrices corresponding to each bandpass effective audio component.

[0020] In one possible implementation, the computing module generates multi-channel feature data based on the multi-channel speech data, specifically by: performing frequency domain transformation on the audio signals in the multi-channel speech data to generate corresponding time-frequency domain matrices; fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data; and generating multi-channel feature data based on the fused data.

[0021] In one possible implementation, the fused data includes an average time-frequency domain matrix or an extended time-frequency domain matrix. The calculation module fuses the time-frequency domain matrices corresponding to at least two audio signals to generate fused data. Specifically, it is used to: average the time-frequency domain matrices corresponding to the at least two audio signals to obtain an average time-frequency domain matrix, wherein the matrix point values ​​in the average time-frequency domain matrix are the average values ​​of the matrix point values ​​at corresponding positions in each of the time-frequency domain matrices; or, concatenate the time-frequency domain matrices corresponding to the at least two audio signals to obtain an extended time-frequency domain matrix.

[0022] In one possible implementation, when the computing module fuses the time-frequency domain matrices corresponding to at least two audio signals to generate fused data, it specifically performs the following: for each matrix point in the time-frequency domain matrix, it creates a corresponding information item, the information item containing the matrix point value of the time-frequency domain matrix corresponding to at least two audio signals at that position, and reference information corresponding to each audio signal, the reference information representing the audio response curve of the audio acquisition unit that acquires the audio signals; and generates fused data based on the information item corresponding to each matrix point.

[0023] Thirdly, embodiments of this application provide an electronic device, including: a memory and a processor;

[0024] The memory stores computer-executed instructions;

[0025] The processor executes computer execution instructions stored in the memory, causing the processor to perform the first aspect and / or various possible implementations of the first aspect as described above.

[0026] Fourthly, embodiments of this application provide a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, are used to implement the first aspect and / or various possible implementations of the first aspect.

[0027] Fifthly, embodiments of this application provide a computer program product, including a computer program that, when executed by a processor, implements the first aspect and / or various possible implementations of the first aspect.

[0028] The reminder voice wake-up method, apparatus, device, storage medium, and program product provided in this application, in response to a voice wake-up operation, acquires multi-channel voice data, which includes at least two audio signals, each independently acquired by a corresponding audio acquisition unit; based on the multi-channel voice data, multi-channel feature data is generated, which characterizes the time-frequency features of the at least two audio signals; the multi-channel feature data is processed by a first model to obtain a voice wake-up word, and a corresponding target function is triggered based on the voice wake-up word. By first determining the wake-up word set by the user based on actual voice wake-up needs, and then processing the acquired multi-channel voice data and inputting it into a pre-trained model, the model determines whether to wake up, thus achieving accurate wake-up and avoiding the problem of false triggering that makes the user feel that the voice wake-up is not intelligent, thereby improving the user experience. Attached Figure Description

[0029] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0030] Figure 1 This is an application scenario diagram of the voice wake-up method provided in the embodiments of this disclosure;

[0031] Figure 2 A flowchart illustrating the voice wake-up method provided in this application;

[0032] Figure 3 for Figure 2 A flowchart illustrating the specific implementation of step S102 in the illustrated embodiment;

[0033] Figure 4 for Figure 3 A flowchart illustrating the specific implementation of step S1022 in the illustrated embodiment;

[0034] Figure 5 for Figure 3 A flowchart illustrating the specific implementation of step S1023 in the illustrated embodiment;

[0035] Figure 6 A flowchart illustrating another implementation of step S102;

[0036] Figure 7 This is a flowchart of step S102B;

[0037] Figure 8 A schematic diagram of the structure of the voice wake-up device provided in this application;

[0038] Figure 9 A schematic diagram of the structure of the electronic device provided in this application.

[0039] The accompanying drawings illustrate specific embodiments of this application, which will be described in more detail below. These drawings and descriptions are not intended to limit the scope of the concept in any way, but rather to illustrate the concept of this application to those skilled in the art through reference to particular embodiments. Detailed Implementation

[0040] Exemplary embodiments will now be described in detail, examples of which are illustrated in the accompanying drawings. When the following description relates to the drawings, unless otherwise indicated, the same numbers in different drawings denote the same or similar elements. The embodiments described in the following exemplary embodiments do not represent all embodiments consistent with this application. Rather, they are merely examples of apparatuses and methods consistent with some aspects of this application as detailed in the appended claims.

[0041] The voice wake-up method provided in this disclosure can be applied to applications (APPs) with artificial intelligence assistant functions, such as voice assistants for electric vehicles and native AI assistant programs in mobile operating systems. More specifically, it can be applied to application scenarios where voice wake-up is used to activate the car's seat massage function or to ask the AI ​​assistant to make a phone call. The executing entity in this embodiment can be a terminal device running the aforementioned AI assistant application, a server deploying the server corresponding to the aforementioned application, or other electronic devices that perform similar functions. When the executing entity is a terminal device, the terminal device executes the method provided in this embodiment by running the aforementioned application. When the executing entity is a server, the server of the aforementioned application with voice wake-up function can run partially or entirely on the server, and the method provided in this embodiment is executed on the server side, while the terminal device runs the client of the application. The communication between the server and the terminal device is based on server-client communication, thereby enabling the terminal device to obtain the execution result of the method provided in this embodiment and display it as needed.

[0042] In some embodiments, the terminal device or server can implement the voice wake-up method provided in this disclosure by running various computer-executable instructions or computer programs. For example, computer-executable instructions can be program-level commands, machine instructions, or software instructions. Computer programs can be native programs or software modules in an operating system; they can be local applications, i.e., programs that need to be installed in the operating system to run, or mini-programs embedded in any APP, i.e., programs that run in a browser environment. In summary, the aforementioned computer-executable instructions can be any form of instruction, and the aforementioned computer programs can be any form of application, module, or plugin; the specific implementation can be configured as needed. Furthermore, in implementing the voice wake-up method provided in this disclosure, the terminal device can execute the method by running computer-executable instructions or computer programs set locally, or by calling computer-executable instructions or computer programs set in an external server. In some embodiments, the server may be an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud storage, cloud communication, cloud database, cloud computing, cloud functions, network services, middleware services, domain name services, security services, content delivery network (CDN), and big data and artificial intelligence platforms. Among these, cloud services may be interactive processing services that can be invoked by terminal devices.

[0043] Figure 1This is an application scenario diagram of the voice wake-up method provided in the embodiments of this disclosure, with reference to... Figure 1 As shown, taking a terminal device (such as a smartphone) as an example, the terminal device is equipped with a sound acquisition unit, such as a microphone. When the terminal device is running a target application with voice wake-up functionality, such as an AI assistant, the user can call out the wake word of the target application. For example, as shown in the figure, the target application has a wake word set to "XX classmate". Then, in response to detecting the user calling the wake word, the target application will respond by playing the sound "I'm here", thereby reminding the user.

[0044] In existing technologies, terminal devices typically acquire audio through a single sound acquisition unit, i.e., a single-channel processing solution. Consequently, this results in low recognition rates, leading to unresponsive calls or false triggers. Existing solutions cannot guarantee the fulfillment of users' actual voice wake-up needs. For example, in everyday voice wake-up scenarios, existing technologies can only wake up the target application in very quiet environments. However, in noisy environments or when the user speaks words similar to the wake-up word, the target application may not respond when the user wants to call it, or it may respond even when the user does not call it. Existing technologies lack front-end signal processing or single-channel processing solutions, resulting in poor flexibility and an inability to guarantee the fulfillment of users' actual voice wake-up needs, thus affecting the user experience.

[0045] The technical solution of this application and how the technical solution of this application solves the above-mentioned technical problems are described in detail below with specific embodiments. These specific embodiments can be combined with each other, and the same or similar concepts or processes may not be described again in some embodiments. The embodiments of this application will now be described with reference to the accompanying drawings.

[0046] Figure 2 A flowchart illustrating the voice wake-up method provided in this application is shown below. Figure 2 As shown, the method includes:

[0047] Step S101: In response to the voice wake-up operation, acquire multi-channel voice data. The multi-channel voice data includes at least two audio signals, and each audio signal is independently acquired by a corresponding audio acquisition unit.

[0048] For example, as referenced Figure 1The illustrated application scenario diagram shows that voice wake-up refers to the user activating a target application by speaking a wake-up word. Acquiring multi-channel voice data refers to the simultaneous acquisition of multiple parallel voice signals containing the same voice event (e.g., speaker's voice) and environmental information through multiple independent audio acquisition devices (such as microphones) in the same time and space. The core characteristics are "multi-channel synchronous acquisition and multi-channel signal association," where each signal can be considered a "channel," and multiple signals together constitute multi-channel voice data. After the user utters the wake-up word, the terminal device responds to the wake-up operation by acquiring multi-channel voice data. The terminal device contains at least two audio acquisition units (i.e., ≥2 channels), such as the two signals acquired by dual microphones at the top and bottom of a mobile phone, or the four signals acquired by microphones next to the four seats in an electric vehicle cabin. Each microphone acquires independently, and different microphones can be independent in physical location and hardware parameters to avoid signal interference and ensure that each signal reflects "sound information from different spatial locations." Essentially, through hardware design and synchronization mechanisms, the "reliability" and "availability" of the data are ensured.

[0049] Step S102: Based on the multi-channel speech data, generate multi-channel feature data, which is used to characterize the time-frequency features of at least two audio signals;

[0050] For example, multi-channel feature data refers to structured data (such as matrices or eigenvectors) that can characterize the "time and frequency characteristics" of each audio signal, while retaining the correlation information between multiple channels. Generating multi-channel feature data based on multi-channel speech data is essentially a data processing step in the voice wake-up process. Specifically, in one possible implementation, the multi-channel speech data acquired in step S101 is processed, segmented, and subjected to time-frequency transformation to obtain the corresponding frequency domain data. Then, the frequency domain data from different time periods are combined to obtain the time-frequency features corresponding to the audio signal.

[0051] Step S103: Process the multi-channel feature data through the first model to obtain the voice wake-up word, and trigger the corresponding target function based on the voice wake-up word.

[0052] For example, the first model processing is, for instance, a pre-trained neural network model that can take sound data as input and identify the content in the sound. This first model can be deployed locally on the terminal device or in the cloud. After obtaining the aforementioned multi-channel speech data, the multi-channel speech data is input into the first model to analyze its features. If its features match a preset wake-up word, a voice wake-up word is generated. Then, based on the voice wake-up word, the corresponding target function is triggered. For example, first, it is identified whether it is the preset wake-up word "XX classmate". Once confirmed as a wake-up word, the corresponding function is immediately triggered, such as opening a specific app, turning on the air conditioner, or activating the seat massage function.

[0053] Furthermore, Figure 3 for Figure 2 The flowchart of the specific implementation of step S102 in the illustrated embodiment shows that, in one possible implementation, step S102 includes:

[0054] Step S1021: Perform blind source separation on the multi-channel speech data to obtain at least two audio components corresponding to the audio signal;

[0055] Step S1022: Obtain the effective audio components corresponding to each audio signal, wherein the effective audio components correspond to human voice signals;

[0056] Step S1023: Generate multi-channel feature data based on the effective audio components.

[0057] For example, the multi-channel voice data in S1021 is multi-source audio recorded by multiple microphones, such as a mixture of sounds from multiple sources including user voice, television sound, and air conditioner noise. After obtaining the multi-channel voice data, blind source separation technology can be used to decompose the multi-source audio into multiple independent audio components (such as "human voice component, television noise component, and air conditioner noise component") without relying on any prior information. Each audio component corresponds to an independent sound source. In S1022, the effective audio components corresponding to each audio signal are obtained, that is, only human voice is retained and noise is removed. The effective audio component is human voice. The filtering logic is as follows: using the blind source separation algorithm, based on the characteristics of different audio components (such as human voice having a fixed frequency range and pitch fluctuations, while noise is chaotic), the audio signal is divided into multiple audio components with different characteristics (frequency characteristics). Then, by filtering, only the audio components that conform to the characteristics of human voice are retained, that is, the effective audio components. Step S1023 is to convert the filtered pure human voice component (effective audio component) into three-dimensional feature data (number of channels * frequency dimension * time dimension) to generate multi-channel feature data.

[0058] Furthermore, Figure 4 for Figure 3The flowchart illustrates the specific implementation of step S1022 in the illustrated embodiment; the specific implementation includes:

[0059] Step S1022A: Extract features from each audio component of the audio signal to obtain the audio features corresponding to the audio components;

[0060] Step S1022B: Load the user's voice model and determine the effective audio components by calculating the feature similarity between the user's voice model and the audio features;

[0061] For example, MFCC / FBANK refers to the characteristics of Mel-Frequency Cepstral Coefficient (MFCC) and Mel Filter Bank (FBANK). It is the core method to convert the original sound waveform into feature data that machines can understand. It is specifically adapted to the human ear's perception of sound and translates the messy sound signal into digital language that the model can analyze.

[0062] For example, feature extraction for each audio component in S1022A refers to extracting information representing the core features of each audio component (such as voiceprint features, frequency patterns, and noise characteristics) to generate a set of feature values, i.e., audio features. After blind source separation in S1021, the terminal device obtains multiple audio components (such as user voice, television noise, and background voice), but cannot specifically identify the content of each audio component. Then, by performing feature extraction on each component separately, audio features are obtained, and different audio components are accurately distinguished based on the audio features, i.e., distinguishing which are user voice, which are background voice, and which are noise. Commonly used features include voiceprint features (higher-order features of MFCC / FBANK) and speech spectrum features.

[0063] Next, in step S1022B, by loading the user's voice model and calculating the feature similarity between the user's voice model and audio features, the audio components that match the user's voice from the multiple audio components are determined, i.e., the effective audio components. Simply put, the effective audio components are compared with the user's voice model, using voiceprint features to find the target voice. This step is the core of accurate screening; it's equivalent to comparing the fingerprint of each voice with a pre-stored user voiceprint template. Loading the user's voice model refers to pre-recorded and trained user voice templates. For example, when a user uses the device for the first time, they record a specific wake word. The device extracts the user's voiceprint features through this wake word and generates a unique user voice model stored locally or in the cloud. Calculating feature similarity involves comparing each audio feature with a human voice model to obtain a feature similarity score. The value range of the feature similarity score is, for example, from 0 to 100. Then, based on a preset similarity threshold (e.g., 80), audio components with feature similarity scores greater than the threshold are determined to be valid audio components (i.e., the user's voice), while all components with similarity scores lower than the threshold are determined to be invalid components and filtered out directly.

[0064] The steps in this embodiment achieve the following beneficial effects: precise prevention of accidental touches, filtering not only noise but also the use of wake words from others (such as accidental calls from family members or wake words from the TV), ensuring that only the user's voice can trigger the function, thus enhancing security; adaptation to personalized scenarios, supporting multi-user devices such as home speakers, where each user has their own unique voice model, triggering only the user's personalized functions after the wake word is spoken (e.g., user A's wake word triggers user A's playlist playback, and user B's wake word triggers user B's playlist playback); and improved robustness in complex scenarios: even when multiple people are speaking simultaneously (such as at a family gathering), the user's voice can be accurately locked without interference.

[0065] Figure 5 for Figure 3 The flowchart of the specific implementation of step S1023 in the illustrated embodiment shows that, in one possible embodiment, the specific implementation of step S1023 includes:

[0066] Step S1023A: Based on the user's voice model, bandpass filtering is performed on the effective audio components to generate bandpass effective audio components;

[0067] For example, bandpass filtering is a signal processing method that preserves specific frequency band components of an audio signal while filtering out other frequency band components. That is, it only allows sound within a specific frequency range to pass through, filtering out all sounds below and above this range (for example, only sounds from 300 Hz to 3500 Hz are allowed, which is the core frequency band of human speech). The filtering bandwidth and filtering center frequency in a bandpass filtering algorithm are the filtering parameters of the algorithm. In this embodiment, the filtering parameters are determined based on the user's voice model; that is, for example, different user voice models correspond to different filtering parameters. In the above steps, firstly, filtering parameters matching the user's voice model are obtained, and then bandpass filtering is performed on the effective audio components based on these filtering parameters to generate bandpass effective audio components. The bandpass effective audio components generated in the above manner are matched with the personalized user voice model. Therefore, the audio features of the bandpass effective audio components are more similar to the user's voiceprint features, thereby improving the accuracy of the multi-channel feature data generated based on the bandpass effective audio components and the recognition rate of voice wake-up words in subsequent steps. Specifically, in this embodiment, bandpass filtering of the effective audio components based on the user voice model is a user-tailored filter. The user voice model records the user's voiceprint-specific frequency range (e.g., a high pitch and a core frequency between 500 Hz and 400 Hz). (0 Hz); Simply put, it's about performing a dedicated frequency filter on the target voice, retaining only the core voiceprint of the wake word. The filtering is based on this dedicated range, making it more accurate than general filtering. This step involves a precise frequency-level purification after filtering out the user's dedicated voice (effective audio components). A frequency filter is customized using a pre-trained user voice model, retaining only the core frequency range of the user's voice that best matches the wake word, while filtering out noise in the effective voice (such as breath sounds or slight hoarseness), making the final voice signal purer. When generating feature data later, the features of the wake word will be more prominent. This step serves three key purposes: 1. Further improve feature purity: By filtering out invalid noise within the human voice, the frequency features of the wake-up word will be more prominent during subsequent feature extraction, leading to more accurate model recognition; 2. Adapt to individual voiceprints and avoid missed recognition: Customized filtering, rather than general filtering, avoids mistakenly filtering the core frequencies of the voiceprint (for example, some people speak in a very low pitch, which general filtering will filter out their low-frequency core, while customized filtering will not), balancing accuracy and adaptability; 3. Reduce the computational burden on subsequent models: After filtering out useless frequency components, the generated feature data will be more focused, and the model will not need to analyze "useless human voice noise," resulting in faster response speed and meeting the requirements of real-time wake-up.

[0068] Step S1023B: Perform frequency domain transformation on the bandpass effective audio components to generate the corresponding time-frequency domain matrix;

[0069] For example, frequency domain conversion of the bandpass effective audio components does not convert the entire audio segment into frequency all at once. Instead, it uses frame-by-frame Fourier transform to analyze small segments. This method better reflects the continuous change characteristic of speech. Frame-by-frame means cutting the entire bandpass effective audio component (i.e., the pure human voice after bandpass filtering) into short segments (e.g., one frame every 20 milliseconds, with adjacent frames overlapping by 10 milliseconds to avoid sound gaps). For example, a 1-second wake word can be cut into about 100 frames. Then, Fourier transform is performed on the above short segments to convert the time-domain sound waveform of this frame into the frequency-domain energy distribution, and the sound energy of each frequency in this frame is calculated.

[0070] The steps in this embodiment involve converting the bandpass effective audio component (pure core human voice) after dedicated filtering from the time domain into a time-frequency domain matrix. The time domain represents the form of sound perceived by humans; as time changes, the volume (amplitude) of the sound changes accordingly. For example, the waveform displayed by a mobile phone recording shows time on the horizontal axis and loudness on the vertical axis. It can only show when the sound is loud and when it is soft, but not the frequency (pitch). The time-frequency domain refers to a two-dimensional dimension that combines time and frequency. It shows both the changes in time and the frequency distribution at each time point. For example, when saying "XY classmate," you can clearly see that the first "X" is low-frequency and high-energy at 0.1 seconds, and the "Y" is mid-frequency and high-energy at 0.3 seconds. Simply put, it turns the sound that changes over time into a two-dimensional map with time on the horizontal axis, frequency on the vertical axis, and color / value representing sound energy. This allows subsequent models to intuitively see which time point and which frequency of sound energy is strong or weak, accurately capturing the syllable and pitch change patterns of the wake-up word.

[0071] Step S1023C: Generate multi-channel feature data based on the time-frequency domain matrix corresponding to each bandpass effective audio component;

[0072] For example, after obtaining the time-frequency domain matrix corresponding to the effective bandpass audio components, the time-frequency domain matrix is ​​stacked by channel according to a unified format to generate multi-channel feature data with channel number * frequency dimension * time dimension. Through the steps of the above embodiments, the effect of simultaneously preserving the spatial differences of multiple microphones and the frequency and timing features of human voice is achieved.

[0073] In another possible implementation Figure 6 The flowchart illustrates another implementation of step S102. For example, a specific implementation of step S102 includes:

[0074] Step S102A: Perform frequency domain transformation on the audio signals in the multi-channel speech data to generate the corresponding time-frequency domain matrix;

[0075] For example, the raw, unprocessed multi-channel speech data (raw audio from all microphones, mixed with noise or human voices) is directly transformed in the frequency domain to generate the corresponding time-frequency domain matrix. Simply put, the messy raw sound is first converted into a frequency-time panorama that can be clearly understood and displayed by the machine, and the raw audio from each microphone is converted into a time-frequency domain format that the model can initially analyze.

[0076] Step S102B: Fuse the time-frequency domain matrices corresponding to at least two audio signals to generate fused data;

[0077] For example, multi-channel original time-frequency domain matrices (one channel for each microphone, with time energy maps corresponding to frequencies of human voice and noise) are fused into multi-channel data. For instance, multiple microphones capturing the same scene's frequency domain panorama are stitched together and weighted according to specific rules to create a fused data panorama. The fusion of the multi-channel time-frequency domain matrices in this embodiment is not a simple numerical addition, but rather a feature-weighted integration between channels based on frequency and time dimensions to meet the real-time and accuracy requirements of voice wake-up. For example, the specific implementation includes: First, energy-weighted fusion, representing the position of each time frame plus a frequency point, calculating the energy value of all channels at that position, and weighting by signal-to-noise ratio (microphone channels with higher signal-to-noise ratios have greater weight, such as the front...). The microphone is closer to the speaker and has less noise, so its weight is set to 0.4; the microphone behind the speaker has more noise, so its weight is set to 0.1. Finally, the fusion energy value at that position is calculated, so that the characteristics of the clear channel dominate and the interference of the noise channel is weakened. Second, mean plus difference fusion, which means that the average energy of all channels at the same time frame and frequency point is calculated first, and then the energy difference value between channels is superimposed. This preserves the common characteristics of multiple channels (the core frequency domain characteristics of all voices / noises) and the spatial differences between channels (to provide a basis for locating the sound source later). Here, the average energy of the time frame and frequency point refers to the core calculation index of multi-channel speech domain fusion. It means that in the time-frequency domain matrix of multiple microphones, for the same time frame and the same frequency point, the arithmetic mean of the sound energy values ​​of all microphone channels at that position is calculated.

[0078] Step S102C: Generate multi-channel feature data based on the fused data.

[0079] For example, after obtaining the fused data through the above steps, the fused data obtained through multi-channel fusion is further converted into C*F*T multi-channel feature data usable by the model, where C represents the number of channels, F represents the frequency, and T represents the time. This step is based on the fused data generated by S102B, and directly performs feature extraction + dimension reconstruction + channel mapping on the fused data to finally generate standardized C*F*T multi-channel feature data.

[0080] Furthermore, C*F*T multi-channel feature data is a standardized three-dimensional feature data format that the model can directly recognize in multi-channel voice wake-up. It is also the final model input data obtained from the original audio after a series of processing steps. By packaging the three core dimensions of sound—spatial information, frequency features, and temporal patterns—into a three-dimensional tensor, the model can simultaneously capture where the sound comes from, the tone / syllable features of the sound, and the order of the sounds, thereby improving the model's accuracy in recognizing wake-up words.

[0081] Figure 7 The flowchart for step S102B is shown below. In one possible implementation, step S102B is specifically implemented as follows:

[0082] The average time-frequency domain matrix is ​​obtained by averaging the time-frequency domain matrices corresponding to at least two audio signals. The matrix point values ​​in the average time-frequency domain matrix are the average values ​​of the matrix point values ​​at corresponding positions in each time-frequency domain matrix. Alternatively, the extended time-frequency domain matrix is ​​obtained by concatenating the time-frequency domain matrices corresponding to at least two audio signals.

[0083] For example, for at least two time-frequency domain matrices, at completely corresponding positions (same time frame and same frequency point), calculate the arithmetic mean of the values ​​at that position of all matrices, and use this average value as the value at the corresponding position of the new matrix. The resulting single two-dimensional matrix is ​​the average time-frequency domain matrix.

[0084] This disclosure also provides another fusion method, namely another implementation of step S102B. In this other possible implementation, the specific implementation of step S102B includes:

[0085] Step S102B-1: For each matrix point in the time-frequency domain matrix, create a corresponding information item. The information item contains the matrix point values ​​of the time-frequency domain matrix corresponding to at least two audio signals at that position, as well as the reference information corresponding to each audio signal. The reference information characterizes the audio response curve of the audio acquisition unit that acquires the audio signals.

[0086] Step S102B-2: Generate fused data based on the information item corresponding to each matrix point.

[0087] In this context, a matrix point is the smallest unit of analysis in the time-frequency domain matrix, determined by a unique time frame and a unique frequency point (for example, a 0.2-second time frame and a 1500 Hz frequency point constitute a matrix point). Reference information refers to the audio response curve of the microphone that acquired the audio, which is an inherent hardware characteristic of the microphone. It records the microphone's sensitivity to different frequencies of sound, calibrates and stores it in advance, and does not change with the acquired signal. The information item of a matrix point is a structure-type data created for a single matrix point. It is the final output of this step, with one information item corresponding to one matrix point, without omissions or repetitions.

[0088] Through the steps of this embodiment, a unique composite and refined information package (information item) can be assigned to each core analysis unit (matrix point) of the multi-channel time-frequency domain matrix; the sound energy values ​​collected by all microphones at the same location and the hardware acquisition characteristics (audio response curve) of each microphone are completely bound together, so that the matrix point, which originally only had a single value, becomes an intelligent data unit with multi-channel energy data and hardware calibration data. This allows subsequent multi-channel fusion, blind source separation and other processing to conform to the actual hardware characteristics of the microphone, avoid signal analysis deviations caused by hardware differences, and improve processing accuracy from the data source level.

[0089] Subsequently, based on the composite information items created for each matrix point in the preceding steps, the multi-channel frequency domain data is finely fused to adapt to hardware characteristics, ultimately generating a single fused data sheet that integrates the core information of multiple channels.

[0090] Figure 8 A schematic diagram of the structure of the voice wake-up device provided in this application is shown in the figure. The voice wake-up device 40 provided in this embodiment includes:

[0091] The acquisition module 401 is used to acquire multi-channel voice data in response to a voice wake-up operation. The multi-channel voice data includes at least two audio signals, and each audio signal is independently acquired by a corresponding audio acquisition unit.

[0092] The calculation module 402 generates multi-channel feature data based on multi-channel speech data. The multi-channel feature data is used to characterize the time-frequency features of at least two audio signals.

[0093] The execution module 403 processes multi-channel feature data through the first model to obtain the voice wake-up word, and triggers the corresponding target function based on the voice wake-up word.

[0094] In one possible implementation, when generating multi-channel feature data, the acquisition module 401 is specifically used to: perform blind source separation on the multi-channel speech data to obtain at least two audio components corresponding to the audio signal; acquire the effective audio components corresponding to each audio signal, wherein the effective audio components correspond to the human voice signal; and generate multi-channel feature data based on the effective audio components.

[0095] In one possible implementation, the acquisition module 401 acquires the effective audio components corresponding to each audio signal, specifically for: extracting features from each audio component of the audio signal to obtain the audio features corresponding to the audio components; loading the user voice model, and determining the effective audio components by calculating the feature similarity between the user voice model and the audio features;

[0096] In one possible implementation, when generating multi-channel feature data, the calculation module 402 is specifically used to: perform bandpass filtering on the effective audio components based on the user's voice model to generate bandpass effective audio components; perform frequency domain transformation on the bandpass effective audio components to generate corresponding time-frequency domain matrices; and generate multi-channel feature data based on the time-frequency domain matrices corresponding to each bandpass effective audio component.

[0097] In one possible implementation, when generating multi-channel feature data, the calculation module 402 is specifically used to: perform frequency domain transformation on the audio signals in the multi-channel speech data to generate a corresponding time-frequency domain matrix; fuse the time-frequency domain matrices corresponding to at least two audio signals to generate fused data; and generate multi-channel feature data based on the fused data.

[0098] In one possible implementation, when generating fused data, the calculation module 402 is specifically used to: average the time-frequency domain matrices corresponding to at least two audio signals to obtain an average time-frequency domain matrix, wherein the matrix point values ​​in the average time-frequency domain matrix are the average values ​​of the matrix point values ​​at corresponding positions in each time-frequency domain matrix; or, concatenate the time-frequency domain matrices corresponding to at least two audio signals to obtain an extended time-frequency domain matrix.

[0099] In one possible implementation, when generating fused data, the calculation module 402 specifically performs the following: for each matrix point in the time-frequency domain matrix, it creates a corresponding information item, which includes the matrix point value of the time-frequency domain matrix corresponding to at least two audio signals at that position, as well as reference information corresponding to each audio signal, and the reference information characterizes the audio response curve of the audio acquisition unit that acquires the audio signals; and generates fused data based on the information item corresponding to each matrix point.

[0100] Figure 9 A schematic diagram of the structure of the electronic device provided in this application. Figure 9 As shown, the electronic device 60 provided in this embodiment includes at least one processor 601 and a memory 602. Optionally, the device 60 further includes a communication component 603. The processor 601, memory 602, and communication component 603 are connected via a bus 604.

[0101] In a specific implementation, at least one processor 601 executes computer execution instructions stored in memory 602, causing at least one processor 601 to perform the above-described method.

[0102] The specific implementation process of processor 601 can be found in the above method embodiments, and its implementation principle and technical effect are similar. It will not be repeated here.

[0103] In the above embodiments, it should be understood that the processor can be a Central Processing Unit (CPU), or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), etc. The general-purpose processor can be a microprocessor or any conventional processor. The steps of the method disclosed in this invention can be directly implemented by a hardware processor, or implemented by a combination of hardware and software modules within the processor.

[0104] The memory may include random access memory (RAM) and may also include non-volatile memory (NVM), such as at least one disk storage device.

[0105] The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of illustration, the buses shown in the accompanying drawings are not limited to a single bus or a single type of bus.

[0106] This application also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0107] This application also provides a computer-readable storage medium storing computer-executable instructions, which, when executed by a processor, implement the above-described method.

[0108] The aforementioned readable storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic storage, flash memory, magnetic disk, or optical disk. The readable storage medium can be any available medium accessible to a general-purpose or special-purpose computer.

[0109] An exemplary readable storage medium is coupled to a processor, enabling the processor to read information from and write information to the readable storage medium. Of course, the readable storage medium can also be a component of the processor. The processor and the readable storage medium can reside in an Application Specific Integrated Circuit (ASIC). Alternatively, the processor and the readable storage medium can exist as discrete components in the device.

[0110] The division of units is merely a logical functional division; in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0111] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0112] In addition, the functional units in the various embodiments of the present invention can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0113] If a function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this invention, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this invention. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0114] Those skilled in the art will understand that all or part of the steps of the above-described method embodiments can be implemented by hardware related to program instructions. The aforementioned program can be stored in a computer-readable storage medium. When executed, the program performs the steps of the above-described method embodiments; and the aforementioned storage medium includes various media capable of storing program code, such as ROM, RAM, magnetic disks, or optical disks.

[0115] Finally, it should be noted that other embodiments of the invention will readily occur to those skilled in the art upon consideration of the specification and practice of the invention disclosed herein. This invention is intended to cover any variations, uses, or adaptations of the invention that follow the general principles of the invention and include common knowledge or customary techniques in the art not disclosed herein, and is not limited to the precise structures described above and shown in the accompanying drawings, and various modifications and changes can be made without departing from its scope. The scope of the invention is limited only by the appended claims.

Claims

1. A voice wake-up method, characterized in that, include: In response to a voice wake-up operation, multi-channel voice data is collected. The multi-channel voice data includes at least two audio signals, and each audio signal is independently collected by a corresponding audio acquisition unit. Based on the multi-channel speech data, multi-channel feature data is generated, which is used to characterize the time-frequency features of at least two audio signals; The multi-channel feature data is processed by the first model to obtain the voice wake-up word, and the corresponding target function is triggered based on the voice wake-up word.

2. The method according to claim 1, characterized in that, Based on the multi-channel speech data, multi-channel feature data is generated, including: Blind source separation is performed on the multi-channel speech data to obtain at least two audio components corresponding to the audio signal; Obtain the effective audio components corresponding to each of the audio signals, wherein the effective audio components correspond to human voice signals; Based on the effective audio components, multi-channel feature data is generated.

3. The method according to claim 2, characterized in that, The step of obtaining the effective audio components corresponding to each of the audio signals includes: Feature extraction is performed on each audio component of the audio signal to obtain the audio features corresponding to the audio components; Load the user's voice model, and determine the effective audio components by calculating the feature similarity between the user's voice model and the audio features.

4. The method according to claim 3, characterized in that, The generation of multi-channel feature data based on the effective audio components includes: Based on the user voice model, bandpass filtering is performed on the effective audio components to generate bandpass effective audio components; The effective bandpass audio components are frequency domain transformed to generate the corresponding time-frequency domain matrix; Multi-channel feature data is generated based on the time-frequency domain matrix corresponding to each of the bandpass effective audio components.

5. The method according to claim 1, characterized in that, The generation of multi-channel feature data based on the multi-channel speech data includes: The audio signals in the multi-channel speech data are frequency domain transformed to generate the corresponding time-frequency domain matrix; The time-frequency domain matrices corresponding to at least two audio signals are fused to generate fused data. Based on the fused data, multi-channel feature data is generated.

6. The method according to claim 5, characterized in that, The fused data includes an average time-frequency domain matrix or an extended time-frequency domain matrix. The process of fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data includes: The average time-frequency domain matrix is ​​obtained by averaging the time-frequency domain matrices corresponding to the at least two audio signals, wherein the matrix point values ​​in the average time-frequency domain matrix are the average values ​​of the matrix point values ​​at corresponding positions in each of the time-frequency domain matrices; or, the extended time-frequency domain matrix is ​​obtained by concatenating the time-frequency domain matrices corresponding to the at least two audio signals.

7. The method according to claim 5, characterized in that, The process of fusing the time-frequency domain matrices corresponding to at least two audio signals to generate fused data includes: For each matrix point in the time-frequency domain matrix, a corresponding information item is created. The information item contains the matrix point values ​​of the time-frequency domain matrix corresponding to at least two audio signals at that position, as well as reference information corresponding to each audio signal. The reference information characterizes the audio response curve of the audio acquisition unit that acquires the audio signals. Based on the information item corresponding to each matrix point, fused data is generated.

8. A voice wake-up function device, characterized in that, include: The acquisition module is used to acquire multi-channel voice data in response to a voice wake-up operation. The multi-channel voice data includes at least two audio signals, and each audio signal is independently acquired by a corresponding audio acquisition unit. The calculation module generates multi-channel feature data based on the multi-channel speech data, the multi-channel feature data being used to characterize the time-frequency features of at least two audio signals; The execution module processes the multi-channel feature data through the first model to obtain the voice wake-up word, and triggers the corresponding target function based on the voice wake-up word.

9. An electronic device, characterized in that, include: Memory, processor; The memory stores computer-executed instructions; The processor executes computer execution instructions stored in the memory, causing the processor to perform the method as described in any one of claims 1-7.

10. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores computer-executable instructions, which, when executed by a processor, are used to implement the method as described in any one of claims 1-7.

Images