Voiceprint coding method, voiceprint coding network training method and electronic device

Abstract

The application provides a voiceprint coding method, a voiceprint coding network training method and an electronic device, and relates to the technical field of natural language processing. The voiceprint coding method comprises the following steps: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; the target mask data is used to represent an application intention of the to-be-processed audio data; the to-be-processed audio data and the target mask data are input into a trained voiceprint coding network, so that the voiceprint coding network infers an encoding result suitable for the application intention based on the to-be-processed audio data and the target mask data. According to the application, the voiceprint coding network can converge better, and the situation that the convergence effect is good or not good for different application intentions can be avoided. For to-be-processed audio data of different application intentions, an encoding result is generated by using the same trained voiceprint coding network, so that the parameter size can be reduced, and storage resources can be saved.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of natural language processing, and particularly relates to a voiceprint coding method, a voiceprint coding network training method and an electronic device. BACKGROUND

[0002] Voiceprint coding is coding of voice. Voiceprint coding can be used to distinguish whether the voice comes from the same person or different persons. In order to meet the requirements of multiple applications such as security, wake-up voiceprint verification, online translation and conference scenarios, two types of voiceprint coding methods are needed: one type is known speech content, such as voice wake-up, the wake-up word is known in advance, and the wake-up word (content known voiceprint) is spoken during registration and verification. The other type is unknown content, and different content is spoken during registration and verification (content unknown voiceprint). Therefore, two types of voiceprint coding neural networks are needed for voiceprint coding processing, which increases the network parameters and the amount of calculation. In addition, the above two types of voiceprint coding usually use two independent neural networks, which are trained with different data and used for inference with two models respectively. In the scenario of smart glasses or resource-limited scenarios, it is actually redundant for storage and calculation. For this reason, a better solution has not been proposed yet. SUMMARY

[0003] Embodiments of the present application provide a voiceprint coding method, a voiceprint coding network training method and an electronic device to solve one or more of the above technical problems.

[0004] In a first aspect, embodiments of the present application provide a voiceprint coding method, comprising: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; wherein the target mask data is used to represent an application intention of the to-be-processed audio data; inputting the to-be-processed audio data and the target mask data into a trained voiceprint coding network, so that the voiceprint coding network infers a coding result suitable for the application intention based on the to-be-processed audio data and the target mask data.

[0005] In a second aspect, an embodiment of the present application provides a voiceprint coding network training method, the voiceprint coding network comprising an audio feature extraction structure, a marking structure and a main network structure; the method comprising: obtaining audio training data and mask data corresponding to the audio training data; wherein the mask data is used to represent the application intention of the audio training data; using the audio feature extraction structure to perform feature extraction on the audio training data to obtain audio feature data; inputting the audio feature data and the mask data into the marking structure in association to obtain marked audio feature data, the marked audio feature data being a feature with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain a voiceprint code; adjusting the voiceprint coding network based on the voiceprint code and a pre-determined loss function to obtain a trained voiceprint coding network.

[0006] In a third aspect, an embodiment of the present application provides an electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to implement any of the above methods.

[0009] Compared with related technologies, the present application has the following advantages:

[0010] The voiceprint coding method, the voiceprint coding network training method and the electronic device provided in the embodiments of the present application, the voiceprint coding method comprises: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; wherein the target mask data is used to represent the application intention of the to-be-processed audio data; inputting the to-be-processed audio data and the target mask data into a trained voiceprint coding network, so that the voiceprint coding network infers an encoding result suitable for the application intention based on the to-be-processed audio data and the target mask data. According to the embodiments of the present application, the corresponding to-be-processed audio data is marked by the target mask data, so as to distinguish the to-be-processed audio data of different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid the situation that the convergence effect of different application intentions is good or not good. The embodiments of the present application generate an encoding result by using the same trained voiceprint coding network for the to-be-processed audio data of different application intentions, which can reduce the parameter size and save storage resources.

[0011] The above description is only a summary of the technical solutions of the present application, in order to more clearly understand the technical means of the present application, the embodiments can be implemented according to the content of the description, and in order to make the above and other purposes, characteristics and advantages of the present application more obvious and easy to understand, the following specific embodiments of the present application are described. BRIEF DESCRIPTION OF DRAWINGS

[0012] In the drawings, the same reference numbers in the several drawings represent the same or similar components or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application, and should not be considered as limiting the scope of the present application.

[0013] Figure 1 A flowchart of a voiceprint coding method provided in the embodiments of the present application is shown;

[0014] Figure 2 A structure diagram of a voiceprint coding neural network provided in the embodiments of the present application is shown;

[0015] Figure 3 A network schematic diagram of a TDNN provided in the embodiments of the present application is shown;

[0016] Figure 4 A network schematic diagram of a D-TDNN provided in the embodiments of the present application is shown;

[0017] Figure 5 A network schematic diagram of a CAM++ provided in the embodiments of the present application is shown;

[0018] Figure 6A possible voiceprint encoding network structure diagram provided in an embodiment of the present application is shown.

[0019] Figure 7 A CAM++ network structure diagram provided in an embodiment of the present application is shown.

[0020] Figure 8 An FCM front-end convolution module structure diagram provided in an embodiment of the present application is shown.

[0021] Figure 9 A data fusion process diagram provided in an embodiment of the present application is shown.

[0022] Figure 10 An optimized training network structure diagram provided in an embodiment of the present application is shown.

[0023] Figure 11 A convolution structure diagram with residual connection provided in an embodiment of the present application is shown.

[0024] Figure 12 A mask-based CAM++ network diagram provided in an embodiment of the present application is shown.

[0025] Figure 13 A voiceprint encoding network training method flowchart provided in an embodiment of the present application is shown.

[0026] Figure 14 An FBANK calculation step provided in an embodiment of the present application is shown.

[0027] Figure 15 A block diagram of an electronic device for implementing an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In the following, only certain exemplary embodiments are simply described. As those skilled in the art can recognize, the described embodiments can be modified in various different ways without departing from the concept or scope of the present application. Therefore, the drawings and the description are considered to be exemplary in nature, rather than limiting.

[0029] To facilitate understanding of the technical solutions of the embodiments of the present application, the related technologies of the embodiments of the present application are described below. The following related technologies can be combined with the technical solutions of the embodiments of the present application in any manner as optional solutions, and all of them belong to the protection scope of the embodiments of the present application.

[0030] First, the voiceprint encoding network that can be used in the present application is described.

[0031] The neural network-based voiceprint coding method has gone through x-vector (a voiceprint coding neural network), TDNN (Time Delay Neural Network), D-TDNN (Dilated Time Delay Neural Network), CAM (Context Aware Masking), and the latest CAM++.

[0032] Figure 2 A structural diagram of a voiceprint coding neural network is shown, Figure 2 The information of Layer, Layercontext, Total context and Input x output is shown, which is used to represent the network structure of a typical voiceprint coding network x-vector. The first 5 layers of x-vector are TDNN, which means that the time dimension view is expanded by using multi-layer NN (Neural Network), that is, longer audio information is contained at the same time, and more complete audio information is referred to for inference. Figure 3 The network structure diagram of TDNN is as follows, Figure 3 The network structure of a four-layer TDNN is shown, and the input and output frames of each layer are represented by a rectangular box. The input of the network is t-13 to t+9, and the output is t. For example, the first layer layer 1 needs 5 frames of data for each calculation, the first 2 frames, the last 2 frames, and the current frame. The output of the first layer is used as the input of the second layer. The second layer layer2, each time 4 frames of data are input, the first 1 frame and the last 2 frames, plus the current frame. The same is true for the output of layer 4.

[0033] Then there is D-TDNN, which considers earlier information on the basis of TDNN and has better effect. As follows Figure 4 In the above, there is FNN (Feed Forward Neural Network) before TDNN, and the output of the feed forward neural network is spliced with the result of TDNN as the input of the next layer.

[0034] The latest voiceprint coding neural network is CAM++, as shown below Figure 5 As shown, it contains FCM (front convolution module) and multiple DenseBlock layers. The FCM is composed of ResBlock (residual block), and the DenseBlock is mainly composed of TDNN and CAM. Figure 5The“Acoustic feature” in the middle represents the input audio feature, and the“ReLU” represents the activation function. The“Global Pooling” represents global pooling, and the“Segment Pooling” represents segment pooling. The“Sigmoid” represents the Sigmoid function, which is a commonly used activation function. Figure 5 The front half of the network is the FCM (Front Convolution Modules), which is used to convert the frequency domain features of a continuous segment of audio in time into feature vectors for subsequent use. There are usually four convolutional layers with residual connections (ResBlock-1 / 2 / 3 / 4 in the figure). Figure 5 The back half of the network is the D-TDNN (Densely connected time delay neural network), and the number of Dense Block layers is variable.

[0035] Voiceprint encoding is the encoding (vector) of sound, and the similarity of the encoding is the similarity of the vector. The voiceprint encoding of each person's speech is different, and the voiceprint encoding corresponding to multiple speeches of each person is similar. The similarity of speeches of different people is smaller. Using this principle, voice encoding (voiceprint) can be used to distinguish whether the sound comes from the same person or different people.

[0036] Smart devices with voice assistants, such as mobile phones, smart glasses, smart watches, smart speakers, etc., basically support voice assistants and voice wake-up. Due to security issues, voice wake-up needs to support voiceprint verification, and only the owner's voice can wake up their own device to prevent the device from being awakened by non-owners. Wake-up voiceprint verification compares the user's spoken wake-up audio with the registered wake-up audio. If the comparison is similar and the similarity is greater than a preset threshold, it is considered to be the owner's wake-up instruction, causing the device to wake up, otherwise, the current wake-up is ignored. In translation scenarios and conference scenarios, it is hoped that the current speaker can be identified, and even the speech content of a specific speaker can be separated from the mixed audio.

[0037] In order to meet the above multiple applications, two types of voiceprint encoding methods are needed: one type is known speech content, such as voice wake-up, where the wake-up word is known in advance, and both registration and verification are spoken wake-up words (content-known voiceprint). The other type is unknown content, where different content is spoken during registration and verification, such as in translation scenarios and conference scenarios, where the separated speech content from the mixed audio is unknown. Therefore, two types of voiceprint encoding neural networks need to be trained for content-known voiceprint and content-unknown voiceprint, respectively, increasing the network parameters and computational complexity.

[0038] The above two voiceprint encodings usually use two independent neural networks, which are respectively trained with different data and infer with two models respectively. In the smart glasses or resource-limited scenarios, it is actually redundant to storage and calculation, but there is no good solution at present.

[0039] Based on this, the present application provides a voiceprint encoding method, a voiceprint encoding network training method and an electronic device. The voiceprint encoding method uses the same neural network to realize two task functions through expansion, which can infer the voiceprint encoding of the audio with known content and the voiceprint encoding of the audio with unknown content, thereby reducing the parameter size and saving storage resources. The same neural network forces to fit two tasks, that is, to learn two types of data, which may not converge or one task is good and the other is not good. In order to better fit, the method can distinguish whether it is a wake-up voiceprint (content known voiceprint) or an unknown content voiceprint by adding a mask, which helps the network to better learn and converge. Even multiple content known voiceprints and content unknown voiceprints can be supported at the same time, as long as the length of the mask is extended.

[0040] The technical solutions of the present application and how the technical solutions of the present application solve the foregoing technical problems will be described in detail below with specific embodiments. Several specific embodiments listed can be combined with each other, and the same or similar concepts or processes can not be described in detail in some embodiments. The embodiments of the present application will be described in detail below with reference to the drawings.

[0041] The embodiment of the present application provides a voiceprint encoding method, which can be applied to smart wearable devices such as smart glasses. As shown in Figure 1 The flowchart of the voiceprint encoding method of an embodiment of the present application is shown, which can include:

[0042] In step S101, the audio data to be processed and the target mask data corresponding to the audio data to be processed are obtained; wherein the target mask data is used to represent the application intention of the audio data to be processed.

[0043] In the embodiments of the present application, the audio data to be processed is audio or frequency domain feature data generated based on natural language of a user, which can be in the form including but not limited to FBANK (Filter Bank, a sound feature extraction method commonly used in speech recognition and audio signal processing) or MFCC (Mel Frequency Cepstral Coefficients). Among them, MFCC is a feature extraction method widely used in speech recognition, speaker recognition and other audio signal processing fields. MFCC aims to simulate the characteristics of the human auditory system, especially in the extraction of spectral features of speech signals, which can effectively capture the timbre information of speech while reducing the data dimension.

[0044] It should be noted that FBANK is the amplitude square sum of the power spectrum of the Mel filter, which omits one step of discrete cosine transform calculation compared with MFCC. The calculation steps of FBANK are shown in Figure 14 As shown. Generally, the audio data is pre-emphasized, the purpose is to enhance the high frequency signal; then frame and window, frame is to separate the audio signal according to 10 ms, window is to prevent spectral leakage, each time 25 ms of signal is used to calculate the feature, that is, each time 10 ms is moved, 25 ms of signal is actually used, and there is 15 ms of historical overlapping information. Then Fourier transform is used to obtain frequency domain signal from time domain signal, that is, frequency spectrum. The frequency domain of a certain time is accumulated to obtain a spectrogram, and then the frequency is mapped to a Mel frequency scale through a Mel filter bank, and finally the FBANK feature is obtained by taking the logarithm. In application, the Mel filter bank generally takes 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0045] The target mask data is an input feature of the voiceprint encoding network, used to represent the application intention of the audio data to be processed. The application intention refers to the goal or purpose that the user hopes to achieve when interacting with the application or system. For example, the application intention includes that the current audio data to be processed is used to wake up the device, or it is hoped that the current audio data to be processed is voiceprint separated. Among them, the current audio data to be processed for waking up the device contains known content, for example, the known content is: stream stream. Among them, it is hoped that the current audio data to be processed is voiceprint separated, for example, the current audio data to be processed includes voiceprints of multiple different people, and the voiceprint of the user of the smart glasses can be separated from the multiple voiceprints. The dimension of the target mask data can be set according to actual needs, which is not limited in the embodiments of the present application, for example, when the dimension is one-dimensional, value 1 can represent that the content is known, and value 0 can represent that the content is unknown.

[0046] It should be noted that in this step, for example, if multiple application intents need to be supported, as long as the dimension of the mask is expanded, for example, 2 dimensions, 4 numbers can be represented, of which 3 represent different known contents and 1 represents unknown content. As follows

[0047] The voiceprint encoding shown in Table 1 supports three wake-up words: Xiaoxi Xiaoxi, Xiaojia Xiaojia, and Hello Jili.

[0048] Mask Class 00 Current network infers unknown content's voiceprint encoding 01 Current network infers known content 1's voiceprint encoding, e.g., "Riverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriverriv 10 ​ 11 ​

[0049] Table 1

[0050] In this step, the corresponding to-be-processed audio data is marked with target mask data, so as to distinguish the to-be-processed audio data of different application intents, which can help the voiceprint encoding network to better learn and converge, and avoid the case that the convergence effect of different application intents is good or bad.

[0051] In step S102, the to-be-processed audio data and the target mask data are input into the trained voiceprint encoding network, so that the voiceprint encoding network infers an encoding result suitable for the application intent based on the to-be-processed audio data and the target mask data.

[0052] In the embodiment of the present application, speech voiceprint encoding, that is, encoding a piece of audio into a vector, the similarity of voiceprint vectors encoded by audios of different people is relatively small, that is, the voiceprints are different, and the similarity of voiceprint vectors obtained by multiple audios of each person is relatively high, that is, the voiceprints are relatively close. The encoding result is a high-dimensional vector obtained by performing speech voiceprint encoding processing on the to-be-processed audio data and the target mask data, for example, 64 / 128 / 256 dimensions.

[0053] The voiceprint encoding network can be a neural network, for example, including but not limited to: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), BERT (Bidirectional Encoder Representations from Transformers), and CAM+ (a kind of voiceprint encoding network), etc. The audio feature is input into a certain neural network to obtain a multi-dimensional vector, for example, a 32 / 64 / 128-dimensional vector. The vector is the voiceprint corresponding to the input audio, and also represents the voice feature of the user. ​ A possible voiceprint encoding network structure diagram is shown. The voiceprint encoding network includes a structure for audio feature extraction and a structure for voiceprint encoding. Specifically, it can be X-vector, TDNN or D-TDNN. ​A network structure diagram of CAM++ is shown, specifically including a structure for audio feature extraction, FCM, and D-TDNN, wherein the FCM includes a plurality of ResBlocks, and the D-TDNN includes a plurality of DenseBlocks.

[0054] In this step, the same trained voiceprint encoding network is used to generate an encoding result for the to-be-processed audio data of different application intents, which can reduce the parameter size and save storage resources.

[0055] The voiceprint encoding method provided in the embodiments of the present application comprises: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; wherein the target mask data is used to represent the application intent of the to-be-processed audio data; inputting the to-be-processed audio data and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network infers an encoding result suitable for the application intent based on the to-be-processed audio data and the target mask data. According to the embodiments of the present application, the target mask data is used to mark the corresponding to-be-processed audio data, thereby distinguishing the to-be-processed audio data of different application intents, which can help the voiceprint encoding network to better learn and converge, and avoid the situation that the convergence effect is good for some application intents and not good for some other application intents. The embodiments of the present application use the same trained voiceprint encoding network to generate an encoding result for the to-be-processed audio data of different application intents, which can reduce the parameter size and save storage resources.

[0056] In a possible implementation, the to-be-processed audio data and the target mask data corresponding to the to-be-processed audio data can be obtained according to the following steps: obtaining the to-be-processed audio data in an unawakened state; performing voice wake-up identification on the to-be-processed audio data; in response to the identification result of the voice wake-up identification being wake-up, determining that the target mask data of the to-be-processed audio data is the mask data corresponding to the wake-up audio; the method further comprises: obtaining a registered voiceprint feature; verifying the encoding result of the voiceprint encoding according to the registered voiceprint feature, and determining whether to perform the wake-up.

[0057] In this possible implementation, the unawakened state refers to a state in which the system, device or model is in a low-power or dormant state, waiting for a specific triggering event to enter the working state. In this state, the system consumes the least resources, and the main purpose is to save energy and resources. The unawakened state is used to obtain the to-be-processed audio data, and the voice awakening recognition is performed on the to-be-processed audio data. The voice awakening recognition is a voice recognition technology used to detect specific trigger words or phrases to awaken the device or start specific functions. If the to-be-processed audio data includes a specific trigger word or phrase, the recognition result of the voice awakening recognition is determined as awakening, and the target mask data of the to-be-processed audio data is determined as the mask data corresponding to the awakening audio, that is, the corresponding relationship between the two is determined, wherein the awakening audio can be used as the data with known content. Based on this step, after the awakening recognition is successful, the voice frame used for the awakening recognition can be added to the mask corresponding to the voiceprint recognition and then input into the voiceprint coding model.

[0058] In this possible implementation, the registered voiceprint feature can also be obtained. The registered voiceprint feature is obtained by collecting and processing the voice samples of the user, extracting and storing the unique voiceprint feature, and used for subsequent identity verification or recognition. The coding result of the voiceprint coding is verified based on the registered voiceprint feature, and it is determined whether to perform the awakening. For example, the registered voiceprint feature and the coding result of the voiceprint coding can be compared, and when the similarity between the two exceeds a preset similarity threshold, it is determined that the awakening is performed, thereby ensuring the security of the awakening operation.

[0059] In one possible implementation, multiple awakening words can be supported, as shown in Table 1. In this case, the voice awakening recognition can be used to determine which awakening word is included in the to-be-processed audio data. Then, the mask data corresponding to the awakening word included in the to-be-processed audio data can be used as the target mask data.

[0060] The steps of the voice awakening voiceprint verification are as follows: 1) User registers the awakening voiceprint: The user can say the awakening word multiple times to obtain multiple voiceprints, and the average value is taken. Alternatively, a long audio containing multiple awakening words is registered, and a voiceprint is calculated. The voiceprint obtained in this step is the registered voiceprint. 2) User awakening: The user says the awakening word to awaken when using. 3) Calculate the voiceprint of the awakening audio: if the voice awakening is successful, the user awakening audio is processed by using the FBANK or MFCC method to extract the sound features, to obtain the to-be-processed audio data, and then the voiceprint is calculated based on the to-be-processed audio data to obtain the awakening voiceprint. 4) Calculate the similarity between the awakening voiceprint and the registered voiceprint. 5) Compare the similarity to determine whether the awakening is passed or not. The calculated similarity is compared with a preset threshold value, and if the threshold value is exceeded, the voiceprint verification is passed, otherwise it is not passed.

[0061] For the same content of voiceprint authentication (verification) scene, it can be realized that in the voice assistant, only the authenticated user can wake up the device. In such a similar scenario, the voiceprint verification needs to register and apply the same content of speech, that is, the audio content spoken by the user during registration, the audio content during training is the same, and other content cannot be guaranteed. Similar to the above-mentioned voiceprint authentication (verification) scene with the same content, in the voice wake-up scene, the wake-up word is known in advance. The advantage of this is that the verification can be made more stringent, the wake-up voiceprint of the registered user is closer, and the distance between the wake-up word voiceprint encoding vectors of the non-registered user and the registered user is larger, that is, the discrimination is larger and it is easier to distinguish. Therefore, the voiceprint application scene with high security requirements can be implemented by using the embodiments of the present application.

[0062] In a possible implementation, the obtaining of the to-be-processed audio data and the target mask data corresponding to the to-be-processed audio data can be performed according to the following steps: obtaining the to-be-processed audio data in a wake-up state; determining the mask data corresponding to the current application scenario as the target mask data corresponding to the to-be-processed audio data; the method further comprises: identifying and / or separating target audio from the mixed audio data according to the encoding result of the voiceprint encoding; wherein the target audio has the same voiceprint feature as the to-be-processed audio data.

[0063] In the embodiments of the present application, the wake-up state means that the system, device or model is in an active working state and can process tasks and perform operations. In this state, the system consumes more resources, but can quickly respond to user requests or process data. The to-be-processed audio data is obtained in the wake-up state. In the wake-up state, the execution subject of the method can be applied to a plurality of different application scenarios, and different mask data is set in advance for different application scenarios. The mask data corresponding to the current application scenario of the execution subject is determined as the target mask data corresponding to the to-be-processed audio data. In this step, the corresponding mask is set for different application scenarios, and the glasses actively select the corresponding mask according to the application scenario.

[0064] In this possible implementation, the target audio can also be identified and / or separated from the mixed audio data according to the encoding result of the voiceprint encoding, wherein the mixed audio data can include a plurality of audios with different voiceprint features. After obtaining the encoding result of the voiceprint encoding, the target audio with the same voiceprint feature as the to-be-processed audio data can be identified and / or separated.

[0065] The following illustrates the voiceprint-based voice separation step: a user registers an audio voiceprint to obtain his own voiceprint, which is irrelevant to the content of speech. In a multi-person speech scene, for example, a person wearing glasses talks to the opposite person, and the microphone is on the glasses, and the recording contains the voices of the glasses wearer and the opposite person. The speech separation neural network is used to separate the speech of the glasses wearer and the opposite person in the mixed audio, and the voiceprint of the glasses wearer is used as a feature to improve the separation effect.

[0066] When a device or apparatus using the method uses a voice assistant, in a voiceprint encoding scene where the content is unknown, the user's speech content can be arbitrary. The voiceprint encoding of the audio obtains a vector, and then the similarity between multiple voiceprints obtained by multiple speech audios is compared. A similarity threshold value is preset, and if it exceeds the threshold value, it is considered to be the same person, otherwise it is different people. This voiceprint encoding usage mainly involves the following two kinds: one is to distinguish who is currently speaking in a multi-person speech scene; the other is an application based on voiceprint separation, wherein the voiceprint encoding can be used as part of the input of the speech separation network.

[0067] In a possible implementation, the voiceprint encoding network includes an audio feature extraction structure, a marking structure, and a main network structure. The voiceprint encoding network is input into the trained voiceprint encoding network based on the to-be-processed audio data and the target mask data to infer an encoding result suitable for the application intent, including: using the audio feature extraction structure to extract features from the to-be-processed audio data to obtain audio feature data; inputting the audio feature data and the mask data into the marking structure to obtain marked audio feature data, the marked audio feature data being features with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain the voiceprint encoding.

[0068] In this possible implementation, in the case where the to-be-processed audio data is speech data, for example, a recording of a user's speech, the audio feature extraction structure can obtain audio feature data based on the to-be-processed audio data, such as FBANK and MFCC. The following explains the process of obtaining FBANK.

[0069] Input: a piece of audio, uninterrupted collection as input. The audio sampling rate is generally 16 kHz, the bit width is 16 bits, and it is single channel. Output: feature value, for example, this piece of audio has 160 ms, and is divided into 16 frames, and each frame feature is 40-dimensional data, so 16 frames of features are [16, 40]. Function: In order to process real-time, the audio is cut into pieces, and generally 10 ms is taken as a frame, and generally 10 ms of audio can extract meaningful features and results. In the subsequent processing process, in order to prevent spectral leakage, the method of partially overlapping the front and rear frames is adopted during processing, and 25 ms of frame length is taken each time, of which 15 ms is historical information, that is, 15 ms of overlap, and only 10 ms is actually moved.

[0070] In the embodiment of the present application, the marking structure includes an FCM front-end convolution module, see ​ The FCM front-end convolution module structure diagram shown in the figure can be composed of multiple layers of ResBlock, including ResBlock1-4, and each ResBlock is composed of convolution and residual connection. ​ A typical convolution structure diagram with residual connection is shown, ​ ​ The association relationship between the convolutions in each ResBlock in ​ After the FCM processing, the multi-frame audio feature data (FBANK or MFCC) is converted into features with time and frequency characteristics. That is, the audio feature data and the mask data are associated and input into the marking structure to obtain marked audio feature data, and the marked audio feature data is a feature with time and frequency characteristics.

[0071] The marking structure can also include a fusion module, which is used to combine multiple features together. There are many fusion methods, including one or more of the following processing: vector splicing, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication and linear combination. For example, features A and B are one-dimensional vectors, and can be fused together in any of the following fusion methods: dimension expansion fusion: [A, B]; addition fusion: A+B, provided that A and B have consistent dimensions; multiplication fusion: A*B. In the embodiment of the present application, dimension expansion fusion is preferably used.

[0072] Finally, the marked audio feature data is input into the main network to obtain the voiceprint code. The network structure of the main network can be selected according to actual needs, and the present application does not make specific limitations on this. Preferably, the main network is a DenseBlock network containing a TDNN (time delay neural network). ​A network diagram of CAM++ is shown. The CAM++ network backend network can be selected as the main network. As shown in ​ The main network includes 3 layers of DenseBlock, and each layer of DenseBlock is composed of FNN (Feedforward Neural Network), TDNN and CAM (Context Aware Masking). Among them, the FNN is a general neural network, such as a convolutional layer or a fully connected layer, composed of multiple layers. The TDNN is a time delay neural network composed of multiple fully connected layers, each layer is staggered in the time dimension to extract information in a large time dimension. The Context in the CAM represents whether the current frame is the target speaker or noise interference. Referring to ​ The network diagram of the mask-based CAM++ is shown. In ​ M is a scalar, which can represent the probability that the current frame is the target speaker. For example, close to 1 represents that the current frame is the target speaker audio, and close to 0 represents that the current frame is interference. M acts as a weight on the result of TDNN. Each layer of TDNN has a CAM. The output of FNN is used as the input of CAM, and M is the output result of the CAM in the figure. The value range of M is 0 to 1, and if the current frame is interference data, the interference data will be filtered out in the case of M being 0. The CAM is used to filter out interference data in the input data of TDNN.

[0073] In a possible implementation, the marking structure includes a fusion module and a front-end convolution module, and the audio feature data and the mask data are associated and input into the marking structure to obtain marked audio feature data, including: inputting the audio feature data and the mask data into the fusion module, and fusing the mask data into the audio feature data by using the fusion module; inputting the audio feature data added with the mask data into the front-end convolution module, and converting the audio feature data fused with the mask data into the marked audio feature data by using the front-end convolution module.

[0074] In this possible implementation, referring to the "multi-task voiceprint encoding network 1" shown in ​ The mask input information can be fused with the audio feature as the input of FCM, that is, the audio feature data and the mask data are input into the fusion module, and the mask data is fused into the audio feature data by using the fusion module. Then, the audio feature data added with the mask data is input into the front-end convolution module, and the audio feature data fused with the mask data is converted into the marked audio feature data by using the front-end convolution module.

[0075] It should be noted that the structure of the voiceprint coding network can be added to the network shown in ​ on the basis of full connection and Softmax. The role of full connection is to map the dimension to the number of categories, for example, 4 categories, and the full connection output is 4-dimensional. The role of Softmax is to make a nonlinear transformation of the values of each category, so that the training is faster and the effect is better. The optimized network structure is shown in ​ .

[0076] In one possible implementation, the marking structure includes a fusion module and a front-end convolution module. The audio feature data and the mask data are associated and input into the marking structure to obtain marked audio feature data, including: inputting the audio feature data into the front-end convolution module, and using the front-end convolution module to convert the audio feature data into a feature extraction result with time and frequency characteristics; inputting the feature extraction result and the mask data into the fusion module, and using the fusion module to fuse the mask data into the feature extraction result to obtain the marked audio feature data.

[0077] In this possible implementation, referring to the "multi-task voiceprint coding network 2" shown in ​ , the information input by the mask can be combined with the output of the FCM as the input of the DenseBlock layer, that is, the audio feature data is input into the front-end convolution module, and the front-end convolution module is used to convert the audio feature data into a feature extraction result with time and frequency characteristics. In the implementation of this step, for example, the input is an audio feature frame (FBANK or MFCC) in a period of time, usually one frame per 25 ms. Assuming that 2 seconds of data are input, there are 80 frames of FBANK features for 2 seconds of audio, and assuming that the FBANK feature is 40-dimensional, the 2-second data is an array of [80, 40]. The output is a feature containing time characteristics and frequency characteristics.

[0078] Then, the feature extraction result and the mask data are input into the fusion module, and the fusion module is used to fuse the mask data into the feature extraction result to obtain the marked audio feature data. In this possible implementation, the calculation amount and the parameter scale are small.

[0079] The embodiment of the application further provides a voiceprint coding network training method. The voiceprint coding network includes an audio feature extraction structure, a marking structure and a main network structure. ​ As shown in the flowchart of the voiceprint coding network training method of the embodiment of the application, the method can include:

[0080] Step S1301: Obtain audio training data and mask data corresponding to the audio training data; wherein, the mask data is used to characterize the application intent of the audio training data.

[0081] In this embodiment, the audio training data is audio data. The mask data is an input feature of the speaker coding network, used to characterize the application intent of the audio training data. Application intent refers to the goal or purpose that a user hopes to achieve when interacting with an application or system. For example, application intent includes whether the content of the currently processed audio data is known or unknown. The dimension of the mask data can be set according to actual needs; this embodiment does not specifically limit this. For example, when the dimension is one-dimensional, a value of 1 can represent known content, and a value of 0 can represent unknown content.

[0082] It should be noted that in this step, for example, if multiple known contents need to be supported, simply expand the dimensions of the mask, such as to 2 dimensions, which can represent 4 numbers, where 3 represent different known contents and 1 represents unknown contents. As shown in Table 1, it supports voiceprint encoding for three wake words: Xiaoxi Xiaoxi, Xiaoji Xiaoji, and Nihao Jili.

[0083] The implementation of this step is illustrated below with a specific example. The audio training data is divided into two types: one with known content and one with unknown content. Because a mask input is added, the mask for known content corresponds to the category of known content, while the mask for unknown content is 0. Taking the three types of voiceprint encoding with known content as an example, the data is organized into the format shown in Table 2 below. The third column is used to characterize whether the input audio content belongs to the unknown content category or the known content category:

[0084]

[0085] Table 2

[0086] Step S1302: Using the audio feature extraction structure, feature extraction is performed on the audio training data to obtain audio feature data.

[0087] In one possible implementation, feature extraction is performed on the audio training data to obtain audio feature data, including: generating FBANK data and / or MFCC data based on the audio training data; and using the FBANK data and / or the MFCC data as audio feature data.

[0088] In the embodiments of the present application, the audio feature data is frequency domain feature data generated based on natural language of a user, which can be in the form of, but not limited to, FBANK (Filter Bank, a sound feature extraction method commonly used in speech recognition and audio signal processing) or MFCC (Mel Frequency Cepstral Coefficients). Among them, MFCC is a feature extraction method widely used in speech recognition, speaker recognition and other audio signal processing fields. MFCC aims to simulate the characteristics of the human auditory system, especially in the extraction of spectral features of speech signals, which can effectively capture the voice quality information while reducing the data dimension.

[0089] It should be noted that FBANK is the amplitude square sum of the power spectrum of the Mel filter, which omits one step of discrete cosine transform calculation compared with MFCC. The calculation steps of FBANK are shown in ​ As shown. Generally, the audio data is pre-emphasized first, the purpose is to enhance the high frequency signal; then frame and window, frame is to separate the audio signal according to 10 ms as a frame, and window is to prevent spectral leakage, each time 25 ms of signal is used to calculate the feature, that is, each time 10 ms is moved, 25 ms of signal is actually used, and there is 15 ms of historical overlapping information. Then, Fourier transform is used to obtain frequency domain signal from time domain signal, that is, frequency spectrum. The frequency domain of a certain time is accumulated to obtain a spectrogram, and then the frequency is mapped to a Mel frequency scale through a Mel filter bank, and finally the FBANK feature is obtained by taking the logarithm. In application, the Mel filter bank generally takes 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0090] In step S1303, the audio feature data and the mask data are associated and input into the mark structure to obtain marked audio feature data, and the marked audio feature data is a feature with time and frequency characteristics.

[0091] In this step, the corresponding audio feature data is marked by using the mask data, so as to distinguish the audio feature data of different application intents, which can help the voiceprint coding network to better learn and converge, and avoid the situation that the convergence effect of different application intents is good or bad.

[0092] In a possible implementation, the marking structure includes a fusion module and a front-end convolution module, an output end of the fusion module is connected with an input end of the front-end convolution module, the audio feature data and the mask data are input into the marking structure in association, to obtain marked audio feature data, including: inputting the audio feature data and the mask data into the fusion module, adding the mask data into the audio feature data by using the fusion module; inputting the audio feature data with the added mask data into the front-end convolution module, extracting time domain data and frequency domain data of the audio feature data with the added mask data by using the front-end convolution module, to obtain the marked audio feature data. Preferably, adding the mask data into the audio feature data includes: performing dimension expansion fusion on the audio feature data and the mask data; wherein the dimension expansion fusion includes one or more of the following processes: vector splicing, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication and linear combination.

[0093] In a possible implementation, the marking structure includes a fusion module and a front-end convolution module, an output end of the front-end convolution module is connected with an input end of the fusion module, the audio feature data and the mask data are input into the marking structure in association, to obtain marked audio feature data, including: inputting the audio feature data into the front-end convolution module, extracting time domain data and frequency domain data of the audio feature data by using the front-end convolution module, to obtain an extraction result; inputting the extraction result and the mask data into the fusion module, adding the mask data into the extraction result by using the fusion module. Preferably, adding the mask data into the extraction result includes: performing dimension expansion fusion on the extraction result and the mask data; wherein the dimension expansion fusion includes one or more of the following processes: vector splicing, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication and linear combination.

[0094] Step S1304: inputting the marked audio feature data into the main network, to obtain a voiceprint code.

[0095] Voiceprint coding is to code a piece of audio into a vector, and the voiceprint vectors coded from audios of different persons have relatively small similarity, that is, the voiceprints are different, and the voiceprint codes obtained from multiple audios of each person have relatively high similarity, that is, the voiceprints are relatively close. The voiceprint code output by the main network is a high-dimensional vector obtained by performing voiceprint coding processing on the to-be-processed audio data and the target mask data, for example, a 64 / 128 / 256-dimensional vector.

[0096] The voiceprint encoding network can be a neural network, for example, including but not limited to: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), BERT (Bidirectional Encoder Representations from Transformers), and CAM+ (Class Activation Mapping++), and the like. The audio features pass through a certain neural network to obtain a multi-dimensional vector, for example, a 32 / 64 / 128-dimensional vector. This vector is the voiceprint corresponding to the input audio, and also represents the voice features of the user. ​ A possible voiceprint encoding network structure diagram is shown. ​ A network structure diagram of CAM++ is shown.

[0097] In a possible implementation, the main network structure includes a plurality of densely connected blocks; the densely connected block includes: a feedforward neural network, a time delay neural network, and a CAM module; wherein the CAM module is used to filter out interference data in the input data of the time delay neural network module.

[0098] In the embodiments of the present application, the network structure of the main network can be selected according to actual needs, which is not limited in the embodiments of the present application. Preferably, the main network is a DenseBlock network containing TDNN (time delay neural network). Referring to ​ The network diagram of CAM++ shown in FIG. 2 can select the CAM++ network backend network as the main network. As shown in ​ The main network contains 3 layers of DenseBlock, and each layer of DenseBlock is composed of FNN (Feedforward Neural Network), TDNN and CAM (Context Aware Masking). Among them, the FNN is a general neural network, such as a convolutional layer or a fully connected layer, which is composed of multiple layers. The TDNN is a time delay neural network, which is composed of multiple fully connected layers, each layer is staggered in the time dimension to extract information in a large time dimension. The Context in the CAM represents whether the current frame is the target speaker or noise interference. Referring to ​ The network diagram of the mask-based CAM++ shown in FIG. 3, in ​M in the figure is a scalar, which can represent the probability that the current frame is spoken by the target speaker, for example, close to 1 represents that the current frame is the target speaker's audio, and close to 0 represents that the current frame is interference. M acts as a weight on the result of TDNN. Each layer of TDNN has a CAM module. The output of FNN is taken as the input of CAM, and M is the output result of the CAM module in the figure. The value range of M is 0 to 1. The CAM module is used to filter out the interference data in the input data of TDNN.

[0099] In this step, the same trained voiceprint encoding network is used to generate voiceprint encodings for audio training data of different application intents, which can reduce the parameter size and save storage resources.

[0100] In step S1405, the voiceprint encoding network is adjusted based on the voiceprint encoding and the predetermined loss function, to obtain a trained voiceprint encoding network.

[0101] In the embodiments of the present application, the loss function can be selected according to actual needs, which is not limited in the embodiments of the present application. For example, the predetermined loss function can include but is not limited to the following types: Euclid Distance-based loss function: Contrastive Loss (Contrastive Loss), Triplet Loss (Triplet Loss), CenterLoss (CenterLoss), etc.; Angular Margin-related loss function: L-Softmax Loss (Large Margin Softmax Loss), A-Softmax Loss (Angular Softmax Loss), CosFace Loss (CosFace Loss), ArcFace Loss (Additive Angular Margin Loss), etc.

[0102] The voiceprint encoding network obtained through the above loss function can achieve the effect of large inter-class vector distance and small intra-class vector distance. The inter-class vector distance is large, that is, the distance of samples of different classes in the feature space should be as large as possible to ensure the discrimination between different classes. This can reduce the confusion between different classes. The intra-class vector distance is small, that is, the distance of samples of the same class in the feature space should be as small as possible to ensure that samples of the same class are clustered together. This helps to improve the similarity of samples of the same class.

[0103] The embodiment of the application provides a voiceprint coding network training method, the voiceprint coding network comprises an audio feature extraction structure, a marking structure and a main network structure; the method comprises the following steps: obtaining audio training data and mask data corresponding to the audio training data; wherein the mask data is used to represent the application intention of the audio training data; using the audio feature extraction structure to perform feature extraction on the audio training data to obtain audio feature data; inputting the audio feature data and the mask data into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain a voiceprint code; based on the voiceprint code and a pre-determined loss function, adjusting the voiceprint coding network to obtain a trained voiceprint coding network. According to the embodiment of the application, the corresponding audio training data is marked by using the mask data, so that the audio training data with different application intentions can be distinguished, which can help the voiceprint coding network to better learn and converge, and avoid the situation that the convergence effect of the audio training data with different application intentions is good or bad. The same voiceprint coding network is used to train the audio training data with different application intentions in the embodiment of the application, which can reduce the parameter size and save storage resources.

[0104] In a possible implementation, before the mask data corresponding to the audio training data is obtained, the method further comprises: determining the number of categories of the content in a training set to which the audio training data belongs; determining the number of mask data in a mask set according to the number of categories, and generating the mask set according to the number of mask data; and obtaining the mask data corresponding to the audio training data comprises: determining the mask data corresponding to the audio training data in the mask set. The number of categories is used to represent the total number of categories of known content and unknown content included in the training set. For example, the training set includes five groups of audio training data with the content of "small stream small stream", which can be "small stream small stream" spoken by different people, the training set further includes eight groups of audio training data with the content of "small record small record", which can be "small record small record" spoken by different people, in addition, the training set further includes one group of audio training data with unknown content, and the training set further includes one group of audio training data with the content of "hello lucky". The number of categories included in the training set is four, that is, the categories of known content are "small stream small stream", "small record small record" and "hello lucky", and the other is the category of unknown content. Based on the number of categories 4, the number of mask data in the mask set should be at least 4, and then a two-dimensional mask set can be determined, and "00", "01", "10" and "11" are used to represent the masks of different categories respectively.

[0105] Corresponding to the application scenario of the voiceprint coding method and the voiceprint coding method provided in the embodiments of the present application, the embodiments of the present application further provide a voiceprint coding device, which can include:

[0106] The acquisition module is configured to acquire the to-be-processed audio data and target mask data corresponding to the to-be-processed audio data, wherein the target mask data is used to represent an application intention of the to-be-processed audio data; and the coding module is configured to input the to-be-processed audio data and the target mask data into a trained voiceprint coding network, so that the voiceprint coding network infers an encoding result suitable for the application intention based on the to-be-processed audio data and the target mask data.

[0107] According to the embodiments of the present application, the target mask data is used to mark the corresponding to-be-processed audio data, so as to distinguish the to-be-processed audio data with different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid the situation that the convergence effect is good for some application intentions and not good for some other application intentions. The embodiments of the present application respectively generate the encoding result for the to-be-processed audio data with different application intentions by using the same trained voiceprint coding network, which can reduce the parameter size and save storage resources.

[0108] In a possible implementation, the acquisition module is specifically configured to: acquire the to-be-processed audio data in an unawakened state; perform voice wake-up identification on the to-be-processed audio data; and in response to an identification result of the voice wake-up identification being wake-up, determine that the target mask data of the to-be-processed audio data is mask data corresponding to wake-up audio. The device further includes an execution module configured to: acquire a registered voiceprint feature; verify the encoding result of the voiceprint coding according to the registered voiceprint feature, and determine whether to perform the wake-up.

[0109] In a possible implementation, the acquisition module is specifically configured to: acquire the to-be-processed audio data in an awakened state; and determine that mask data corresponding to a current application scenario is the target mask data corresponding to the to-be-processed audio data. The device further includes a processing module configured to: identify and / or separate target audio from mixed audio data according to the encoding result of the voiceprint coding, wherein the target audio has the same voiceprint feature as the to-be-processed audio data.

[0110] In a possible implementation, the voiceprint encoding network comprises an audio feature extraction structure, a marking structure and a main network structure, and the encoding module is specifically configured to: extract features of the audio data to be processed by using the audio feature extraction structure to obtain audio feature data; input the audio feature data and the mask data into the marking structure to obtain marked audio feature data, the marked audio feature data being features with time and frequency characteristics; and input the marked audio feature data into the main network to obtain the voiceprint encoding.

[0111] In a possible implementation, the marking structure comprises a fusion module and a front-end convolution module, and the encoding module is specifically configured to: input the audio feature data and the mask data into the fusion module to fuse the mask data into the audio feature data by using the fusion module; and input the audio feature data to which the mask data is added into the front-end convolution module to convert the audio feature data to which the mask data is added into the marked audio feature data by using the front-end convolution module.

[0112] In a possible implementation, the marking structure comprises a fusion module and a front-end convolution module, and the encoding module is specifically configured to: input the audio feature data into the front-end convolution module to convert the audio feature data into feature extraction results with time and frequency characteristics by using the front-end convolution module; input the feature extraction results and the mask data into the fusion module to fuse the mask data into the feature extraction results by using the fusion module to obtain the marked audio feature data.

[0113] Corresponding to the application scenarios of the voiceprint encoding network training method and the voiceprint encoding network training method provided in the embodiments of the present application, the embodiments of the present application further provide a voiceprint encoding network training apparatus, the voiceprint encoding network comprising an audio feature extraction structure, a marking structure and a main network structure, which can comprise:

[0114] a data module configured to obtain audio training data and mask data corresponding to the audio training data, wherein the mask data is used to represent an application intention of the audio training data; an extraction module configured to extract features of the audio training data by using the audio feature extraction structure to obtain audio feature data; an association module configured to input the audio feature data and the mask data into the marking structure in association to obtain marked audio feature data, the marked audio feature data being features with time and frequency characteristics; an input module configured to input the marked audio feature data into the main network to obtain a voiceprint encoding; and a training module configured to adjust the voiceprint encoding network based on the voiceprint encoding and a pre-determined loss function to obtain a trained voiceprint encoding network.

[0115] According to the embodiments of this application, the corresponding audio training data is labeled with mask data, thereby distinguishing audio training data with different application intentions. This can help the voiceprint coding network learn and converge better, avoiding situations where there is no convergence or the convergence effect of different application intentions is good and bad. The embodiments of this application train the same voiceprint coding network for audio training data with different application intentions, which can reduce the parameter scale and save storage resources.

[0116] In one possible implementation, the data module is specifically used for: determining the number of categories of the content in the training set where the audio training data is located; determining the number of mask data in the mask set based on the number of categories; generating a mask set based on the number of mask data; and determining the mask data corresponding to the audio training data in the mask set.

[0117] The functions of each module in each device in the embodiments of this application can be found in the corresponding description in the above method, and they have corresponding beneficial effects, which will not be repeated here.

[0118] ​ This is a block diagram of an electronic device used to implement embodiments of this application. For example... ​ As shown, the electronic device includes a memory 1501 and a processor 1502. The memory 1501 stores a computer program that can run on the processor 1502. When the processor 1502 executes the computer program, it implements the method described in the above embodiments. The number of memories 1501 and processors 1502 can be one or more.

[0119] The electronic device also includes:

[0120] The communication interface 1503 is used to communicate with external devices and perform data exchange and transmission.

[0121] If the memory 1501, processor 1502, and communication interface 1503 are implemented independently, they can be interconnected via a bus to communicate with each other. This bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus, or an Extended Industry Standard Architecture (EISA) bus, etc. This bus can be divided into address bus, data bus, control bus, etc. For ease of representation, ​ The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0122] Optionally, if the memory 1501, the processor 1502 and the communication interface 1503 are integrated on a chip, the memory 1501, the processor 1502 and the communication interface 1503 can complete the communication among each other through an internal interface.

[0123] The embodiment of the present application provides a computer readable storage medium, which stores a computer program, and the program is executed by a processor to implement the method provided in the embodiment of the present application.

[0124] The embodiment of the present application provides a computer program product, wherein the computer program product comprises a computer program, and the computer program is executed by a processor to implement the method provided in the embodiment of the present application.

[0125] The embodiment of the present application further provides a chip, which comprises a processor, is used for calling and running instructions stored in a memory, and makes a communication device installed with the chip execute the method provided in the embodiment of the present application.

[0126] The embodiment of the present application further provides a chip, which comprises an input interface, an output interface, a processor and a memory, and the input interface, the output interface, the processor and the memory are connected through an internal connection path, and the processor is used for executing code in the memory, and when the code is executed, the processor is used for executing the method provided in the embodiment of the present application.

[0127] It should be understood that the processor described above can be a central processing unit (CPU), and can also be other general-purpose processors, digital signal processors (DSP), application specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. The general-purpose processor can be a microprocessor or any conventional processor, etc. It should be noted that the processor can be a processor supporting an advanced RISC machine (ARM) architecture.

[0128] Further, the memory can optionally include a read-only memory and a random access memory. The memory can be a volatile memory or a non-volatile memory, or can include both volatile and non-volatile memory. The non-volatile memory can include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically EPROM (EEPROM), or a flash memory, among others. The volatile memory can include a random access memory (RAM) used as an external cache memory. By way of example and not limitation, many forms of RAM are available. For example, a static random access memory (SRAM), a dynamic random access memory (DRAM), a synchronous dynamic random access memory (SDRAM), a double data rate SDRAM (DDR SDRAM), an enhanced SDRAM (ESDRAM), a Sync link DRAM (SLDRAM), and a direct Rambus RAM (DR RAM) can be used.

[0129] In the above embodiments, the implementation can be wholly or partially realized through software, hardware, firmware or any combination thereof. When realized by software, the implementation can be wholly or partially realized in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed by a computer, the computer instructions generate the flow or function according to the present disclosure wholly or partially. The computer can be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions can be stored in a computer readable storage medium or transmitted from one computer readable storage medium to another.

[0130] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data for analysis, stored data, displayed data, etc.) involved in the present disclosure are all information and data authorized by the user or authorized by all parties, and the collection, use and processing of related data need to comply with relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation portal for user to choose authorization or refusal.

[0131] In the description of the application, the description of the terms "one embodiment", "some embodiments", "an example", "a specific example", or "some examples" etc. means that the specific features, structures, materials or characteristics described in connection with the embodiment or example are included in at least one embodiment or example of the application. Moreover, the specific features, structures, materials or characteristics described can be combined in any appropriate manner in one or more embodiments or examples. In addition, different embodiments or examples described in the specification and characteristics of different embodiments or examples can be combined and combined by those skilled in the art without contradiction.

[0132] In addition, the terms "first", "second" are only for descriptive purposes and cannot be understood as indicating or implying relative importance or implicitly indicating the number of the indicated technical features. Therefore, the features defined with "first", "second" can explicitly or implicitly include at least one of the features. In the description of the application, the meaning of "multiple" is two or more, unless otherwise specifically limited.

[0133] Any process or method described in the flowchart or otherwise described herein can be understood as representing code modules, segments, or portions of code that include one or more executable instructions for implementing specific logic functions or steps. And the scope of the preferred embodiments of the application includes additional implementation in which the functions can be performed in an order different from that shown or discussed, including functions performed in substantially simultaneous fashion or in reverse order according to the functions involved.

[0134] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as a list of executable instructions for implementing the logic function, which can be specifically implemented in any computer readable medium for instruction execution system, device or equipment (such as computer based system, system including processor or other system that can take instructions from instruction execution system, device or equipment and execute instructions) or in conjunction with these instructions execution system, device or equipment.

[0135] It should be understood that parts of the application can be implemented in hardware, software, firmware or a combination thereof. In the above described embodiments, a plurality of steps or methods can be implemented by software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above described embodiment method can be instructed by the relevant hardware through a program, which can be stored in a computer readable storage medium, and the program includes one or a combination of steps of the method embodiment when executed.

[0136] In addition, each of the function units in each embodiment of the present application can be integrated in one processing module, or each unit can be physically present separately, or two or more units can be integrated in one module. The integrated module can be realized in the form of hardware or in the form of a software function module. When the integrated module is realized in the form of a software function module and sold or used as an independent product, it can also be stored in a computer readable storage medium. The storage medium can be a read-only memory, a magnetic disk or an optical disk, etc.

[0137] The above is only exemplary embodiments of the present application, but the protection scope of the present application is not limited thereto, and any person skilled in the art can easily think of various changes or replacements within the technical scope disclosed in the present application, which should be covered within the protection scope of the present application. Therefore, the protection scope of the present application should be subject to the protection scope of the claims.

Claims

1. A voiceprint encoding method, comprising: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; wherein the target mask data is used to represent an application intention of the to-be-processed audio data; inputting the to-be-processed audio data and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network infers an encoding result suitable for the application intention based on the to-be-processed audio data and the target mask data; the voiceprint encoding network comprises an audio feature extraction structure, a marking structure and a main network structure, and inputting the to-be-processed audio data and the target mask data into the trained voiceprint encoding network so that the voiceprint encoding network infers an encoding result suitable for the application intention based on the to-be-processed audio data and the target mask data comprises: using the audio feature extraction structure to extract features of the to-be-processed audio data to obtain audio feature data; inputting the audio feature data and the mask data into the marking structure to obtain marked audio feature data, the marked audio feature data being features with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain the voiceprint encoding.

2. The method of claim 1, wherein, obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data, comprising: obtaining the to-be-processed audio data in an unawakened state; performing voice wake-up recognition on the to-be-processed audio data; in response to a recognition result of the voice wake-up recognition being wake-up, determining that the target mask data of the to-be-processed audio data is mask data corresponding to wake-up audio. The method further comprises: obtaining a registered voiceprint feature; verifying the encoding result of the voiceprint encoding according to the registered voiceprint feature to determine whether to perform the wake-up.

3. The method of claim 1, wherein, obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data, comprising: obtaining the to-be-processed audio data in an awakened state; determining that mask data corresponding to a current application scenario is the target mask data corresponding to the to-be-processed audio data. The method further comprises: identifying and / or separating target audio from mixed audio data according to the encoding result of the voiceprint encoding; wherein the target audio has the same voiceprint feature as the to-be-processed audio data.

4. The method of claim 1, wherein, The marking structure comprises a fusion module and a front-end convolution module, and inputting the audio feature data and the mask data into the marking structure to obtain the marked audio feature data comprises: inputting the audio feature data and the mask data into the fusion module, and using the fusion module to fuse the mask data into the audio feature data; inputting the audio feature data added with the mask data into the front-end convolution module, and using the front-end convolution module to convert the audio feature data fused with the mask data into the marked audio feature data.

5. The method of claim 1, wherein, The marking structure comprises a fusion module and a front-end convolution module, and inputting the audio feature data and the mask data into the marking structure to obtain the marked audio feature data comprises: input the audio feature data into the front-end convolution module, and convert the audio feature data into a feature extraction result with time and frequency characteristics by using the front-end convolution module; input the feature extraction result and the mask data into the fusion module, and fuse the mask data into the feature extraction result by using the fusion module to obtain the labeled audio feature data. 6.A voiceprint encoding network training method, the voiceprint encoding network comprising an audio feature extraction structure, a labeling structure and a main network structure; the method comprising: obtaining audio training data and mask data corresponding to the audio training data; wherein the mask data is used to represent the application intention of the audio training data; extracting features of the audio training data by using the audio feature extraction structure to obtain audio feature data; associating the audio feature data and the mask data to input the labeling structure to obtain labeled audio feature data, wherein the labeled audio feature data is a feature with time and frequency characteristics; inputting the labeled audio feature data into the main network to obtain a voiceprint code; adjusting the voiceprint encoding network based on the voiceprint code and a pre-determined loss function to obtain a trained voiceprint encoding network; the trained voiceprint encoding network is used to infer the voiceprint code suitable for the application intention of the to-be-processed audio data. 7.An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method of any one of claims 1-6 when executing the computer program. 8.A computer readable storage medium, wherein the computer readable storage medium stores a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-6.

9. A computer program product, wherein, The computer program product comprises a computer program, and the computer program is executed by a processor to implement the method of any one of claims 1-6.

Images