Voice wake-up data processing method and device and storage medium

Abstract

The invention relates to a voice wake-up data processing method and device and a storage medium. The voice wake-up data processing method comprises the following steps: collecting first false wake-up data, and converting the first false wake-up data into a first spectrogram; the first spectrogram is processed through a generative adversarial network, a second spectrogram is generated, the second spectrogram comprises second false wake-up data, and the number of the second false wake-up data is higher than that of the first false wake-up data. According to the invention, the false wake-up rate of the whole wake-up process is reduced.

Description

Technical Field

[0001] The present disclosure relates to the field of voice wake-up, and in particular to a method, device and storage medium for processing voice wake-up data. Background Art

[0002] The voice wake-up of smart devices generally includes the first-level wake-up and the second-level wake-up. The first-level wake-up is keyword detection, and the second-level wake-up is voiceprint detection. The two correspond to different voice wake-up models. Wake-up can only pass the second level after passing the first level. The false wake-up rate of the first level wake-up will affect the false wake-up rate of the entire wake-up process.

[0003] In the related technology, when training the voice wake-up model for the first-level wake-up, the relevant parameters are optimized from the model features, so that the model can learn the features ignored in the general model training process. The ignored features will cause feature escape, thereby increasing the false wake-up rate of the first-level wake-up voice and indirectly increasing the false wake-up rate of the entire wake-up process. Summary of the invention

[0004] In order to overcome the problems existing in the related art, the present disclosure provides a method, device and storage medium for processing voice wake-up data.

[0005] According to a first aspect of an embodiment of the present disclosure, a method for processing voice wake-up data is provided, including: collecting first false wake-up data, and converting the first false wake-up data into a first spectrum graph; processing the first spectrum graph through a generative adversarial network to generate a second spectrum graph, wherein the second spectrum graph includes second false wake-up data, and the number of the second false wake-up data is higher than the number of the first false wake-up data.

[0006] In one implementation, the method further includes: training a first wake-up network based on the second false wake-up data.

[0007] In one embodiment, the processing of the first spectrum map through a generative adversarial network to generate a second spectrum map includes: extracting features of the first spectrum map layer by layer based on a loss function through an encoding path of the generative adversarial network to obtain spectrum feature maps of different depth levels; and processing the deepest feature map of the spectrum feature maps of different depth levels through a decoding path of the generative adversarial network to obtain the second spectrum map.

[0008] In one embodiment, the method also includes: obtaining detailed information of other spectral feature maps other than the deepest spectral feature map among the spectral feature maps of different depth levels; based on a semantic enhancement mechanism, processing multiple spectral feature maps with the deepest levels among the spectral feature maps of different depth levels to obtain semantic information.

[0009] In one embodiment, the deepest feature map among the spectral feature maps of different depth levels is processed through the decoding path of the generative adversarial network, including: adding the detail information to the deepest feature map layer by layer based on the upsampling mechanism of the generative adversarial network, and adding the semantic information to the deepest feature map layer by layer based on the semantic mechanism of the generative adversarial network.

[0010] In one embodiment, the loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss, and the pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined in the following manner: the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding to the weight, and the product of the relative adversarial loss corresponding to the weight is determined as the loss function.

[0011] According to a second aspect of an embodiment of the present disclosure, a device for processing voice wake-up data is provided, including: a collection unit, used to collect first false wake-up data and convert the first false wake-up data into a first spectrum graph; a processing unit, used to process the first spectrum graph through a generative adversarial network to generate a second spectrum graph, wherein the second spectrum graph includes second false wake-up data, and the number of the second false wake-up data is higher than the number of the first false wake-up data.

[0012] In one implementation, the processing unit is further configured to: train a first wake-up network based on the second false wake-up data.

[0013] In one embodiment, the processing unit processes the first spectrum map through a generative adversarial network to generate a second spectrum map in the following manner: through the encoding path of the generative adversarial network, the features of the first spectrum map are extracted layer by layer based on the loss function to obtain spectrum feature maps of different depth levels; through the decoding path of the generative adversarial network, the deepest feature map among the spectrum feature maps of different depth levels is processed to obtain the second spectrum map.

[0014] In one embodiment, the processing unit is also used to: obtain detailed information of other spectral feature maps other than the deepest spectral feature map among the spectral feature maps of different depth levels; based on the semantic enhancement mechanism, process multiple spectral feature maps with the deepest levels among the spectral feature maps of different depth levels to obtain semantic information.

[0015] In one embodiment, the processing unit processes the deepest feature map in the spectral feature maps of different depth levels through the decoding path of the generative adversarial network in the following manner: based on the upsampling mechanism of the generative adversarial network, the detail information is added to the deepest feature map layer by layer, and based on the semantic mechanism of the generative adversarial network, the semantic information is added to the deepest feature map layer by layer.

[0016] In one embodiment, the loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss, and the pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined by the processing unit in the following manner: the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding to the weight, and the product of the relative adversarial loss corresponding to the weight is determined as the loss function.

[0017] According to a third aspect of an embodiment of the present disclosure, a device for processing voice wake-up data is provided, comprising: a processor; and a memory for storing processor executable instructions; wherein the processor is configured to: execute the method for processing voice wake-up data described in any one of the embodiments of the first aspect or the second aspect.

[0018] According to a fourth aspect of an embodiment of the present disclosure, a storage medium is provided, in which instructions are stored. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method for processing voice wake-up data described in any one of the embodiments of the first aspect or the second aspect.

[0019] The technical solution provided by the embodiments of the present disclosure may include the following beneficial effects: collecting false awakening data in actual scenarios, processing the false awakening data by generating adversarial networks, generating false awakening data that meets the awakening model training requirements, and training the first-level awakening network. Through the present disclosure, the false awakening rate of the entire awakening process is reduced.

[0020] It is to be understood that the foregoing general description and the following detailed description are exemplary and explanatory only and are not restrictive of the present disclosure. BRIEF DESCRIPTION OF THE DRAWINGS

[0021] The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the present disclosure and, together with the description, serve to explain the principles of the present disclosure.

[0022] Figure 1 The figure is a flowchart of a method for processing voice wake-up data according to an exemplary embodiment.

[0023] Figure 2 The figure is a flow chart of a method for generating a second spectrum diagram according to an exemplary embodiment.

[0024] Figure 3 is a flow chart of a method for generating a second frequency spectrum diagram according to yet another exemplary embodiment.

[0025] Figure 4 The figure is a flowchart of a method for processing a deepest feature map according to an exemplary embodiment.

[0026] Figure 5 It is a block diagram of a method for obtaining false wake-up data and performing first-level wake-up training according to an exemplary embodiment of the present disclosure.

[0027] Figure 6 It is a block diagram of a method for obtaining a primary wake-up model based on a generative adversarial network according to an exemplary embodiment of the present disclosure.

[0028] Figure 7 The present invention is a block diagram of a device for processing voice wake-up data according to an exemplary embodiment.

[0029] Figure 8 The invention is a block diagram of a device for processing voice wake-up data according to an exemplary embodiment. DETAILED DESCRIPTION

[0030] Here, exemplary embodiments will be described in detail, examples of which are shown in the accompanying drawings. When the following description refers to the drawings, unless otherwise indicated, the same numbers in different drawings represent the same or similar elements. The implementations described in the following exemplary embodiments do not represent all implementations consistent with the present disclosure.

[0031] The method for processing voice wake-up data provided by the embodiments of the present disclosure is applied to a scenario in which first-level wake-up training is performed based on the wake-up data to obtain a first-level wake-up model.

[0032] The voice wake-up of the terminal is divided into primary wake-up and secondary wake-up. The primary wake-up is keyword detection, which wakes up the terminal based on the pre-recorded wake-up word. The secondary wake-up is voiceprint detection, which determines whether the voiceprint characteristics of the wake-up sound match the voiceprint characteristics of the owner who recorded the wake-up word. After both primary wake-up and secondary wake-up are passed, the entire wake-up process is completed. When the terminal is voice-wake-up, after receiving the wake-up word, there is a time sequence relationship between the primary wake-up and the secondary wake-up. The wake-up can only pass the secondary after passing the primary wake-up. Therefore, the false wake-up rate of the primary wake-up will affect the false wake-up rate of the secondary wake-up, and then affect the false wake-up rate of the entire wake-up process.

[0033] The voice wake-up of the terminal is implemented based on the pre-trained language model and acoustic model. The language model is used to resolve the ambiguity of the wake-up word so that the system can understand the user's voice command more accurately. By using a larger language model and introducing contextual information, the false wake-up rate (FWR) can be reduced. The acoustic model is the core part of the voice wake-up system, which is used to identify the target wake-up word pre-recorded by the user and distinguish the target wake-up word from background noise. By collecting more data and continuous training, the acoustic model can be continuously improved, and the accuracy and stability of the acoustic model can be improved, thereby reducing the false wake-up rate.

[0034] In the related art, when training the voice wake-up model for the first-level wake-up, the wake-up words recorded by the user are used as the positive layer, and the background noise and interference sound are used as negative samples, and the voice wake-up model is trained based on the positive and negative samples. Due to the limited collection of background noise and interference sound, the negative samples used to train the voice wake-up model are insufficient, resulting in the optimization of relevant parameters from the model features during the training process, and the inability to effectively exclude the features that serve as interference items, so that the model learns the interference features that are generally ignored in the model training process. The ignored interference features will cause feature escape, causing the wake-up model to be falsely awakened when the interference features are identified, thereby increasing the false awakening rate of the first-level wake-up voice, and indirectly increasing the false awakening rate of the entire wake-up process.

[0035] In view of this, the present disclosure proposes a method for processing voice wake-up data, which collects false wake-up data of a low order of magnitude in real scenarios, processes the false wake-up data through a generative adversarial network, and generates false wake-up data of a high order of magnitude, thereby meeting the wake-up model training requirements, and trains a first-level wake-up network based on a sufficient amount of false wake-up data to obtain a first-level wake-up model, reduce the first-level false wake-up rate, and reduce the false wake-up rate of the entire wake-up process.

[0036] Figure 1 FIG. 1 is a flow chart of a method for processing voice wake-up data according to an exemplary embodiment. Figure 1 As shown, the method includes step S101 to step S102.

[0037] In step S101 , first false awakening data is collected and converted into a first frequency spectrum.

[0038] In step S102, the first spectrum graph is processed by a generative adversarial network to generate a second spectrum graph.

[0039] The second spectrum diagram includes second false awakening data, and the amount of the second false awakening data is greater than the amount of the first false awakening data.

[0040] It is understandable that when the wake-up method is to determine the voiceprint, some false wake-up data that were not identified in the first-level wake-up process can be identified based on the user's voiceprint features, but the second-level wake-up will also have feature escape, and all false wake-up data cannot be identified. Therefore, the false wake-up rate of the first-level wake-up will also affect the false wake-up rate of the second-level wake-up, and affect the false wake-up rate of the entire wake-up process.

[0041] In the disclosed embodiment, the first false awakening data is data collected in an actual scene, including false awakening data such as noise and background sound as interference items, and the data judged as false awakening by the secondary awakening can be screened out by script. The acquired false awakening data is usually time domain data, expressed as a digital sequence, including different time points and the amplitude of the sound at the corresponding time points. The collected false awakening data expressed as time domain data is subjected to Fourier transform, and the time domain data can be converted into frequency domain data, that is, the amplitude information of the sound at each frequency. Furthermore, the frequency and amplitude of the frequency domain data are represented in a linear or logarithmic manner, and the frequency domain and amplitude are plotted through a drawing library (such as Matplotlib), and a spectrum diagram containing the frequency domain wake-up data, that is, a first spectrum diagram, can be obtained.

[0042] In the embodiment of the present disclosure, a second spectrogram is generated based on the first spectrogram by using a generative adversarial network (GAN). The purpose of using the generative adversarial network is as follows:

[0043] 1. The false awakening data set obtained through actual collection is limited, resulting in insufficient negative sample data for first-level awakening, which is not enough to control the feature escape of first-level awakening. In order to obtain enough false awakening data as negative samples and control the feature escape of the first-level awakening model, a large amount of false awakening data is generated based on the actually collected false awakening data using a generative adversarial network. A generative adversarial network is a generative model that includes a generator and a discriminator, which can learn data distribution and generate new data samples. During the training process, the generator network of the generative adversarial network learns the actually collected false awakening data and generates new false awakening data, while the discriminator network is responsible for judging whether the generated samples are consistent with the real data. The two networks compete and cooperate with each other, so that the generator network can generate more and more realistic false awakening data.

[0044] 2. The generative adversarial network performs well in identifying image features and can accurately identify the spectral features of the first spectrogram generated based on the actual collected false wake-up data. The generative adversarial network is a model composed of a generator network and a discriminator network. The generator network learns the input first spectrogram and generates new image samples, while the discriminator network is responsible for distinguishing new image samples from real images. During the training process, the two networks compete and cooperate with each other and iterate continuously, so that the generator network can generate more and more realistic image samples, and the discriminator network can also distinguish the generated samples from the real images more accurately. In this way, the generative adversarial network can learn more details and features of the data, so as to better identify the features of the picture and generate the second spectrogram.

[0045] In the example at the beginning of this disclosure, the second spectrogram is a spectrogram generated by a generative adversarial network based on the first spectrogram. The generative adversarial network is a network model specifically used to process images. It can generate a large number of images that are similar to but different from the existing images based on the existing images. Therefore, the order of magnitude of the second spectrogram is much higher than that of the first spectrogram. It can be understood that the second spectrogram also contains audio information, and the spectrogram can be converted into frequency domain data or time domain data by extraction and conversion. In summary, a large amount of false wake-up data can be generated based on the generative adversarial network and the small amount of false wake-up data collected.

[0046] In the embodiment of the present disclosure, after obtaining a sufficient amount of false wake-up data, the false wake-up data can be used as negative samples, and the collected wake-up data (such as the wake-up words recorded by the user) can be used as positive samples to perform first-level wake-up training to obtain the first wake-up model. It can be understood that based on the generative adversarial network, the present disclosure has obtained a sufficient amount of false wake-up data as negative samples, so the first-level training model obtained by training has an accurate recognition rate, and can accurately identify wake-up data and false wake-up data, thereby reducing the false wake-up rate of the first-level wake-up, and reducing the false wake-up rate of the second-level wake-up, thereby reducing the false wake-up rate of the entire wake-up process. The following embodiments of the present disclosure illustrate the training method of the wake-up network.

[0047] In one implementation of the disclosed embodiment, the method further includes: training the first wake-up network based on the second false wake-up data.

[0048] In an exemplary embodiment of the present disclosure, the following method is used to obtain training data and perform primary network training:

[0049] 1. Data collection: Collect user voice data as positive samples and false wake-up data (background sound, noise, etc.) as negative samples.

[0050] 2. Feature extraction: Use Mel Frequency Cepstral Coefficients (MFCC) to extract features from speech data.

[0051] 3. Data preprocessing: Perform preprocessing operations such as normalization and noise reduction on the extracted features to improve the accuracy of the model.

[0052] 4. Model training: Use the training data set to train the deep learning model so that the deep learning model can recognize the keywords issued by the user and the interference sounds other than the keywords issued by the user, thereby obtaining the first-level wake-up model.

[0053] In the embodiment of the present disclosure, the generative adversarial network used to generate the second spectrogram is a convolutional neural network including an encoding path (Ecoding Path) and a decoding path (Ecoding Path), which can extract the features of the first spectrogram actually collected, and perform feature supplementation on the first spectrogram after the features are extracted, thereby obtaining the second spectrogram. The following embodiment of the present disclosure describes the process of generating the second spectrogram by processing the first spectrogram based on the generative adversarial network.

[0054] Figure 2 FIG. 1 is a flow chart of a method for generating a second spectrum diagram according to an exemplary embodiment. Figure 2 As shown, the method includes step S201 to step S202.

[0055] In step S201, the encoding path of the generative adversarial network is used to extract the features of the first spectrum map layer by layer based on the loss function to obtain spectrum feature maps of different depth levels.

[0056] In step S202, the deepest feature map among the spectrum feature maps at different depth levels is processed by generating a decoding path of the adversarial network to obtain a second spectrum map.

[0057] In the disclosed embodiment, the features of the first spectrum map are extracted hierarchically through the coding path of the generative adversarial network to obtain multiple images of different depth levels and different sizes. That is, the first spectrum map passes through the coding path, and the length and width of the feature map generated in the coding path are continuously reduced. And in the process of extracting image features in the coding path, the number of feature map channels is increased by increasing the number of convolution kernels. It can be understood that as the number of convolution layers continues to increase, shallow features such as color texture of the first spectrum map are gradually extracted, and the information contained in the deepest feature map finally obtained by the coding channel is mainly semantic information.

[0058] In the disclosed embodiment, the deepest feature map is filled with feature information through a decoding path symmetrical to the encoding path, the feature map size is increased, and the feature map channels are reduced, so that a second spectrum map of the same family as the first spectrum map but different from the first spectrum map can be obtained. It can be understood that by supplementing different feature information for the deepest feature map, the second spectrum map has different shallow information, and a large number of different second spectrum feature maps can be obtained, thereby obtaining a large amount of second false wake-up data.

[0059] In the disclosed embodiment, features of the first spectrum graph are extracted through the coding path of the generative adversarial network, and features are supplemented through the coding path of the generative adversarial network to obtain a large number of second spectrum graphs, thereby obtaining a large amount of second false awakening data, so that the first awakening training has sufficient negative samples, thereby solving the problem of high false awakening rate caused by insufficient negative samples during awakening training.

[0060] In the embodiment of the present disclosure, when upsampling the deepest feature map in the decoding path to enrich the image features, it is necessary to use the semantic features and detail information of the first spectrum map extracted by the encoding path. The following embodiment of the present disclosure further illustrates the process of the encoding path of the generative adversarial network processing the first spectrum map.

[0061] Figure 3 FIG. 4 is a flow chart of a method for generating a second spectrum diagram according to another exemplary embodiment. Figure 3 As shown, the method includes step S301 to step S302.

[0062] In step S301, detailed information of other spectral feature maps except the deepest spectral feature map among spectral feature maps of different depth levels is obtained.

[0063] In step S302, based on the semantic enhancement mechanism, multiple spectrum feature maps with the deepest levels among the spectrum feature maps with different depth levels are processed to obtain semantic information.

[0064] In the disclosed embodiment, while extracting the first spectrum map features and obtaining feature maps of different depth levels, detailed information of each level of images except the deepest image is also extracted to supplement the features of the feature maps of the corresponding level in the decoding path. In addition, the scale semantic information of each level of images except the deepest image is also extracted, and the deep semantic information is extracted through the semantic enhancement mechanism (SIR) based on the last two layers of feature maps. Corresponding to the acquisition of scale semantic information and deep semantic information, the scale semantic information and deep semantic information are combined into semantic information (Infos). The semantic information is used to supplement the deepest feature map step by step in combination with the convolution kernel.

[0065] In the disclosed embodiment, the decoding path of the generative adversarial network extracts the first spectrum graph features step by step, and also extracts detail information and semantic information for feature supplementation of the encoding path.

[0066] In the embodiment of the present disclosure, after obtaining the deepest feature map, detail information and semantic information for feature supplementation through the encoding path, the detail information and semantic information are supplemented to the deepest feature map respectively through the decoding path of the generative adversarial network, and other supplementary information is supplemented to the deepest feature map to obtain a second spectrum map. The following embodiment of the present disclosure describes the method of supplementing detail information and semantic information to the deepest feature map.

[0067] Figure 4 FIG. 1 is a flowchart of a method for processing a deepest feature map according to an exemplary embodiment. Figure 4 As shown, the method includes step S401 to step S402.

[0068] In step S401, semantic information and detail information are obtained.

[0069] In step S402, the detail information is added layer by layer to the deepest feature map based on the upsampling mechanism of the generative adversarial network, and the semantic information is added layer by layer to the deepest feature map based on the semantic mechanism of the generative adversarial network.

[0070] In the disclosed embodiment, the acquired detail information includes information corresponding to feature maps other than the deepest feature map in the encoding path. Based on the symmetrical structure of the encoder and decoder in the generative adversarial network, the detail information of feature maps at different levels in the encoder is correspondingly supplemented to the feature maps at the corresponding levels in the decoder.

[0071] In the disclosed embodiment, the acquired semantic information is added to the first layer of convolution kernel (corresponding to the deepest feature map) in the decoder through a semantic supplementation mechanism (SIS), and it is iterated synchronously with the convolution kernel based on the semantic supplementation mechanism, and the semantic information is supplemented to the feature map of the corresponding level in each iteration.

[0072] In the disclosed embodiment, in addition to semantic information and detail information, the decoder also adds other feature information to each layer of feature map based on an upsampling mechanism.

[0073] In the disclosed embodiment, the above-mentioned detail information and other feature information need to be further processed by the residual channel attention mechanism (RCBA) and pixel reorganization (Pixel Shuffle) of the decoding path before they can be added to the feature map, increase the feature map information, and expand the feature map size. Semantic information can be directly added to the corresponding feature map through the convolution kernel.

[0074] In the disclosed embodiment, a decoder of a generative adversarial network is used to supplement information to the deepest feature map, thereby obtaining a second spectrum map whose authenticity is close to that of the actual image, and obtaining a sufficient amount of false wake-up data.

[0075] In the embodiment of the present disclosure, the loss function (Leaky Relu) is determined by multiple losses and the weight corresponding to each loss. The following embodiment of the present disclosure illustrates the loss function in the encoding path of the generative adversarial network.

[0076] In one implementation of the disclosed embodiment, the loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss. The pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined in the following manner: the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding to the weight, and the product of the relative adversarial loss corresponding to the weight is determined as the loss function.

[0077] In the disclosed embodiment, the pixel-by-pixel loss function L MSE As shown below:

[0078]

[0079] Among them, Bi is the input random spectrum image, G θG To generate the network, I i It is the standard error wake-up spectrum diagram, w is generally the width, and h is generally the height.

[0080] In the embodiment of the present disclosure, if only pixel-by-pixel loss is used during training, serious artifacts will result. To this end, a more effective low-perceptual loss L is added to the loss function. percep Low perceptual loss L percep Map the image to the features before the activation function, rather than the features after the activation function, and the expression is:

[0081]

[0082] in is the 3_1 layer parameter before the VGG-16 network activation function (Relu), Bi is the input random spectrum image, G θG To generate the network, I i It is the standard error wake-up spectrum diagram, w is generally the width, and h is generally the height.

[0083] In the disclosed embodiment, the generative adversarial network forces the generated image to be indistinguishable from the real image based on the concept of "adversarial loss". However, it is found through experiments that the optimization process on the standard model often leads to mode collapse. Therefore, the relative adversarial loss is added by utilizing the characteristics of relative adversarial (RALSGAN). As a term in the loss function, its formula is as follows:

[0084]

[0085] Among them G θG represents the generator, D θD represents the discriminator. There are two discriminators in this paper. These two discriminators are respectively for pictures of different scales, which can guide the generator to produce better results. Pdata(x) refers to the sample X is sampled from the real data, E x～Pdata(x) refers to the expectation of samples X sampled from real data, It refers to the expected value of data randomly generated by Z from a false distribution, and D(G(z)) refers to putting a false sample Z into the generator G to generate a sample and then putting it into the discriminator to identify the sample.

[0086] In the disclosed embodiment, each loss constituting the loss function has a corresponding weight (hyperparameter), and the loss function is expressed as follows:

[0087]

[0088] Among them, each weight α, β, γ can be adaptively adjusted based on demand.

[0089] In an exemplary embodiment of the present disclosure, the loss function is:

[0090]

[0091] In the disclosed embodiment, the loss function for extracting image features is determined based on the relative adversarial loss, pixel-by-pixel loss, and self-induction loss based on the weights corresponding to each loss, so as to avoid serious artifacts when generating a spectrogram and prevent the generative adversarial network from collapsing. It is ensured that a second feature map close to the real image is extracted and the stability of the generative adversarial network is guaranteed.

[0092] In the disclosed embodiment, various laboratory parameters for performing and feature training can be adaptively adjusted. In an exemplary embodiment, various laboratory parameters for performing and feature training are set as follows: the data set is 10GB of false wake-up data collected by dot-marking. The batch size used for training is 8, and the original image size is 96×128 random / false wake-up image pairs. The learning rate of all layers is initialized to 1×104, and is reduced by 10 times every 5×105 iterations, for a total of 100 iterations. For optimization, an optimization (Adam Optimizer) with a momentum of 0.9 and a weight decay of 1×104 is used. All experiments were performed on a graphics card (NVIDIA 2080Ti) using a framework (PyTorch).

[0093] In an exemplary embodiment of the present disclosure, Figure 5 The following method is used to obtain false awakening data and generate a first-level training model:

[0094] 1. Collect false wake-up data, and filter out the data that is judged as false wake-up by secondary wake-up through script. After obtaining the sound data, convert it into frequency domain data through Fourier transform, and draw the frequency domain data into a spectrum diagram.

[0095] 2. Generate adversarial network, use the small amount of collected spectrogram data to train the generative adversarial network.

[0096] 3. Data with false wake-up features, generate a large amount of spectrum graph data through generative adversarial networks.

[0097] 4. First-level awakening training: A large number of data sets with false awakening features generated by generative adversarial networks are added as negative samples to train the first-level awakening network.

[0098] 5. Low false awakening rate model, obtain a low false awakening rate model.

[0099] In an exemplary embodiment of the present disclosure, Figure 6 The block diagram of the method for obtaining a first-level wake-up model based on a generative adversarial network is shown in FIG. The process of generating a network from inputting a random spectrum graph to outputting a false wake-up feature spectrum graph is as follows:

[0100] 1. A random real spectrum image first passes through the encoding path (extraction path), and the features of the spectrum image are extracted layer by layer in the encoding path to obtain feature maps at different levels. The length and width of the generated feature map are continuously reduced, and the number of feature map channels increases with the increase of the convolution kernel. When generating the feature map, the generated feature map will also be saved for subsequent intermediate layers to supplement the detailed information. In response to the generation of the last two feature maps, the semantic information is generated by processing the last two layers of feature maps through the semantic enhancement mechanism.

[0101] 2. The last layer of feature maps enters the decoding path (expansion path). In the decoding path, the feature maps are supplemented with detail information and other feature information through upsampling, residual channel attention mechanism and pixel reorganization. The length and width of the feature maps continue to increase, and the number of channels decreases as the number of convolution kernels decreases. While generating a large-scale feature map, the features saved in the decoding path are supplemented into the feature map as detail information, and semantic information (Infos) is also supplemented into the feature map. After three upsamplings, a spectrum map with false awakening features is finally generated.

[0102] 3. The discriminator discriminates the spectrum graph generated by the generator based on the real spectrum graph, and continuously judges the similarity between the generated spectrum graph and the real spectrum graph until the authenticity of the generated spectrum graph cannot be distinguished.

[0103] In an embodiment of the present disclosure, real false wake-up data is collected and converted into a spectrum graph. The converted spectrum graph is input into a generative adversarial network, and the generative adversarial network generates a large number of spectrum graphs containing false wake-up data based on the input spectrum graph. In response to obtaining a sufficient amount of false wake-up data, a first-level wake-up training is performed based on the false wake-up data to obtain a first-level wake-up model. Through the present disclosure, the actual false wake-up data is processed by a generative adversarial network, a sufficient amount of false wake-up data is obtained, and a first-level wake-up model with a low false wake-up rate is obtained, the first-level false wake-up rate is reduced, and the second-level false wake-up rate is reduced, thereby reducing the false wake-up rate of the entire wake-up process.

[0104] Based on the same concept, the embodiment of the present disclosure also provides a device 100 for processing voice wake-up data.

[0105] It is understandable that the voice wake-up data processing device 100 provided in the embodiment of the present disclosure includes hardware structures and / or software modules corresponding to the execution of each function in order to realize the above functions. In combination with the units and algorithm steps of the examples disclosed in the embodiments of the present disclosure, the embodiments of the present disclosure can be implemented in the form of hardware or a combination of hardware and computer software. Whether a function is executed in the form of hardware or computer software driving hardware depends on the specific application and design constraints of the technical solution. Those skilled in the art may use different methods to implement the described functions for each specific application, but such implementation should not be considered to exceed the scope of the technical solution of the embodiments of the present disclosure.

[0106] Figure 7 is a block diagram of a device 100 for processing voice wake-up data according to an exemplary embodiment. Figure 7 The device includes a collection unit 101 and a processing unit 102.

[0107] The collecting unit 101 is used to collect first false awakening data and convert the first false awakening data into a first frequency spectrum diagram.

[0108] The processing unit 102 is used to process the first spectrum graph through a generative adversarial network to generate a second spectrum graph.

[0109] The second spectrum diagram includes second false awakening data, and the amount of the second false awakening data is greater than the amount of the first false awakening data.

[0110] In one implementation, the processing unit 102 is further configured to: train the first wake-up network based on the second false wake-up data.

[0111] In one embodiment, the processing unit 102 processes the first spectrum map through a generative adversarial network to generate a second spectrum map in the following manner: extract the features of the first spectrum map layer by layer based on the loss function through the encoding path of the generative adversarial network to obtain spectrum feature maps of different depth levels. Process the deepest feature map of the spectrum feature maps of different depth levels through the decoding path of the generative adversarial network to obtain the second spectrum map.

[0112] In one embodiment, the processing unit 102 is further configured to obtain detailed information of other spectral feature maps other than the deepest spectral feature map among spectral feature maps of different depth levels. Based on the semantic enhancement mechanism, multiple spectral feature maps of the deepest level among spectral feature maps of different depth levels are processed to obtain semantic information.

[0113] In one embodiment, the processing unit 102 processes the deepest feature map in the spectral feature maps of different depth levels through the decoding path of the generative adversarial network in the following manner: based on the upsampling mechanism of the generative adversarial network, the detail information is added to the deepest feature map layer by layer, and based on the semantic mechanism of the generative adversarial network, the semantic information is added to the deepest feature map layer by layer.

[0114] In one embodiment, the loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss, and the pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined by the processing unit 102 in the following manner: the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding to the weight, and the product of the relative adversarial loss corresponding to the weight is determined as the loss function.

[0115] Regarding the device in the above embodiment, the specific manner in which each module performs operations has been described in detail in the embodiment of the method, and will not be elaborated here.

[0116] Figure 8 1 is a block diagram of a device 200 for processing voice wake-up data according to an exemplary embodiment. The device 200 may be provided as a terminal. For example, the device 200 may be a mobile phone, a computer, a digital broadcast terminal, a messaging device, a game console, a tablet device, a medical device, a fitness device, a personal digital assistant, etc.

[0117] Reference Figure 8 , the device 200 may include one or more of the following components: a processing component 202 , a memory 204 , a power component 206 , a multimedia component 208 , an audio component 210 , an input / output (I / O) interface 212 , a sensor component 214 , and a communication component 216 .

[0118] The processing component 202 generally controls the overall operation of the device 200, such as operations associated with display, phone calls, data communications, camera operations, and recording operations. The processing component 202 may include one or more processors 220 to execute instructions to perform all or part of the steps of the above-described method. In addition, the processing component 202 may include one or more modules to facilitate interaction between the processing component 202 and other components. For example, the processing component 202 may include a multimedia module to facilitate interaction between the multimedia component 208 and the processing component 202.

[0119] The memory 204 is configured to store various types of data to support operations on the device 200. Examples of such data include instructions for any application or method operating on the device 200, contact data, phone book data, messages, pictures, videos, etc. The memory 204 can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as static random access memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic disk or optical disk.

[0120] The power component 206 provides power to the various components of the device 200. The power component 206 may include a power management system, one or more power supplies, and other components associated with generating, managing, and distributing power for the device 200.

[0121] The multimedia component 208 includes a screen that provides an output interface between the device 200 and the user. In some embodiments, the screen may include a liquid crystal display (LCD) and a touch panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from the user. The touch panel includes one or more touch sensors to sense touch, slide, and gestures on the touch panel. The touch sensor may not only sense the boundaries of the touch or slide action, but also detect the duration and pressure associated with the touch or slide operation. In some embodiments, the multimedia component 208 includes a front camera and / or a rear camera. When the device 200 is in an operating mode, such as a shooting mode or a video mode, the front camera and / or the rear camera may receive external multimedia data. Each front camera and rear camera may be a fixed optical lens system or have a focal length and optical zoom capability.

[0122] The audio component 210 is configured to output and / or input audio signals. For example, the audio component 210 includes a microphone (MIC), and when the device 200 is in an operation mode, such as a call mode, a recording mode, and a speech recognition mode, the microphone is configured to receive an external audio signal. The received audio signal can be further stored in the memory 204 or sent via the communication component 216. In some embodiments, the audio component 210 also includes a speaker for outputting audio signals.

[0123] I / O interface 212 provides an interface between processing component 202 and peripheral interface modules, such as keyboards, click wheels, buttons, etc. These buttons may include but are not limited to: a home button, a volume button, a start button, and a lock button.

[0124] The sensor assembly 214 includes one or more sensors for providing various aspects of the status assessment of the device 200. For example, the sensor assembly 214 can detect the open / closed state of the device 200, the relative positioning of components, such as the display and keypad of the device 200, the sensor assembly 214 can also detect the position change of the device 200 or a component of the device 200, the presence or absence of user contact with the device 200, the orientation or acceleration / deceleration of the device 200 and the temperature change of the device 200. The sensor assembly 214 can include a proximity sensor configured to detect the presence of a nearby object without any physical contact. The sensor assembly 214 can also include an optical sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 214 can also include an accelerometer, a gyroscope sensor, a magnetic sensor, a pressure sensor or a temperature sensor.

[0125] The communication component 216 is configured to facilitate wired or wireless communication between the device 200 and other devices. The device 200 can access a wireless network based on a communication standard, such as WiFi, 2G or 3G, or a combination thereof. In an exemplary embodiment, the communication component 216 receives a broadcast signal or broadcast-related information from an external broadcast management system via a broadcast channel. In an exemplary embodiment, the communication component 216 also includes a near field communication (NFC) module to facilitate short-range communication. For example, the NFC module can be implemented based on radio frequency identification (RFID) technology, infrared data association (IrDA) technology, ultra-wideband (UWB) technology, Bluetooth (BT) technology and other technologies.

[0126] In an exemplary embodiment, the apparatus 200 may be implemented by one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field programmable gate arrays (FPGAs), controllers, microcontrollers, microprocessors or other electronic components to perform the above method.

[0127] In an exemplary embodiment, a non-transitory computer-readable storage medium including instructions is also provided, such as a memory 204 including instructions, and the instructions can be executed by the processor 220 of the device 200 to perform the above method. For example, the non-transitory computer-readable storage medium can be a ROM, a random access memory (RAM), a CD-ROM, a magnetic tape, a floppy disk, an optical data storage device, etc.

[0128] It is to be understood that in the present disclosure, "plurality" refers to two or more than two, and other quantifiers are similar. "And / or" describes the association relationship of associated objects, indicating that three relationships may exist. For example, A and / or B can represent: A exists alone, A and B exist at the same time, and B exists alone. The character " / " generally indicates that the associated objects before and after are in an "or" relationship. The singular forms "a", "the" and "the" are also intended to include plural forms, unless the context clearly indicates other meanings.

[0129] It is further understood that the terms "first", "second", etc. are used to describe various information, but such information should not be limited to these terms. These terms are only used to distinguish the same type of information from each other, and do not indicate a specific order or degree of importance. In fact, the expressions "first", "second", etc. can be used interchangeably. For example, without departing from the scope of the present disclosure, the first information can also be referred to as the second information, and similarly, the second information can also be referred to as the first information.

[0130] It will be further understood that the terms “center”, “longitudinal”, “lateral”, “front”, “back”, “up”, “down”, “left”, “right”, “vertical”, “horizontal”, “top”, “bottom”, “inside”, “outside”, etc., indicating orientations or positional relationships based on the orientations or positional relationships shown in the accompanying drawings, are only for the convenience of describing the present embodiment and simplifying the description, and do not indicate or imply that the device or element referred to must have a specific orientation, be constructed and operated in a specific orientation.

[0131] It can be further understood that, unless otherwise specified, “connection” includes a direct connection without other components between the two, and also includes an indirect connection with other components between the two.

[0132] It is further understood that, although the operations are described in a specific order in the drawings in the embodiments of the present disclosure, it should not be understood as requiring the operations to be performed in the specific order shown or in a serial order, or requiring the execution of all the operations shown to obtain the desired results. In certain environments, multitasking and parallel processing may be advantageous.

[0133] Those skilled in the art will readily appreciate other embodiments of the present disclosure after considering the specification and practicing the invention disclosed herein. The present disclosure is intended to cover any variations, uses or adaptations of the present solution, which follow the general principles of the present disclosure and include common knowledge or customary technical means in the art that are not disclosed in the present disclosure.

[0134] It should be understood that the present disclosure is not limited to the precise structures that have been described above and shown in the drawings, and that various modifications and changes may be made without departing from the scope thereof. The scope of the present disclosure is limited only by the scope of the appended claims.

Claims

1. A method for processing voice wake-up data, characterized in that: include: Collecting first false awakening data, and converting the first false awakening data into a first frequency spectrum; The first spectrum graph is processed by a generative adversarial network to generate a second spectrum graph, wherein the second spectrum graph includes second false wake-up data, and the amount of the second false wake-up data is higher than the amount of the first false wake-up data.

2. The method according to claim 1, characterized in that: The method further comprises: The first wake-up network is trained based on the second false wake-up data.

3. The method according to claim 1, characterized in that The step of processing the first spectrum graph by generating an adversarial network to generate a second spectrum graph includes: Extracting features of the first spectrum graph layer by layer based on a loss function through an encoding path of the generative adversarial network to obtain spectrum feature graphs at different depth levels; The deepest feature map among the frequency spectrum feature maps at different depth levels is processed by generating a decoding path of the adversarial network to obtain the second frequency spectrum map.

4. The method according to claim 3, characterized in that The method further comprises: Obtaining detailed information of other frequency spectrum feature maps except the deepest frequency spectrum feature map among the frequency spectrum feature maps of different depth levels; Based on the semantic enhancement mechanism, a plurality of spectrum feature maps with the deepest levels among the spectrum feature maps with different depth levels are processed to obtain semantic information.

5. The method according to claim 4, characterized in that The processing of the deepest feature map among the spectral feature maps of different depth levels by generating a decoding path of the adversarial network includes: Based on the upsampling mechanism of the generative adversarial network, the detail information is added layer by layer to the deepest feature map, and based on the semantic mechanism of the generative adversarial network, the semantic information is added layer by layer to the deepest feature map.

6. The method according to claim 3, characterized in that The loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss, and the pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined in the following way: The loss function is determined as the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding weight, and the product of the relative adversarial loss corresponding weight.

7. A device for processing voice wake-up data, characterized in that: include: A collection unit, configured to collect first false awakening data and convert the first false awakening data into a first frequency spectrum diagram; A processing unit is used to process the first spectrum graph through a generative adversarial network to generate a second spectrum graph, wherein the second spectrum graph contains second false wake-up data, and the amount of the second false wake-up data is higher than the amount of the first false wake-up data.

8. The device according to claim 7, characterized in that The processing unit is also used for: The first wake-up network is trained based on the second false wake-up data.

9. The device according to claim 7, characterized in that The processing unit processes the first spectrogram by generating an adversarial network in the following manner to generate a second spectrogram: Extracting features of the first spectrum graph layer by layer based on a loss function through an encoding path of the generative adversarial network to obtain spectrum feature graphs at different depth levels; The deepest feature map among the frequency spectrum feature maps at different depth levels is processed by generating a decoding path of the adversarial network to obtain the second frequency spectrum map.

10. The device according to claim 9, characterized in that The processing unit is also used for: Obtaining detailed information of other frequency spectrum feature maps except the deepest frequency spectrum feature map among the frequency spectrum feature maps of different depth levels; Based on the semantic enhancement mechanism, a plurality of spectrum feature maps with the deepest levels among the spectrum feature maps with different depth levels are processed to obtain semantic information.

11. The device according to claim 10, characterized in that The processing unit processes the deepest feature map among the spectral feature maps at different depth levels through a decoding path of a generative adversarial network in the following manner: Based on the upsampling mechanism of the generative adversarial network, the detail information is added layer by layer to the deepest feature map, and based on the semantic mechanism of the generative adversarial network, the semantic information is added layer by layer to the deepest feature map.

12. The device according to claim 9, characterized in that The loss function includes pixel-by-pixel loss, self-induction loss and relative adversarial loss, and the pixel-by-pixel loss, self-induction loss and relative adversarial loss correspond to different weights respectively. The loss function is determined by the processing unit in the following manner: The loss function is determined as the sum of the product of the pixel-by-pixel loss and the corresponding weight, the product of the self-induction loss corresponding weight, and the product of the relative adversarial loss corresponding weight.

13. A device for processing voice wake-up data, characterized in that: include: processor: a memory for storing processor-executable instructions; The processor is configured to: execute the method for processing voice wake-up data according to any one of claims 1 to 6.

14. A storage medium, characterized in that: The storage medium stores instructions. When the instructions in the storage medium are executed by a processor, the processor is enabled to execute the method for processing voice wake-up data according to any one of claims 1 to 6.

Images