Voiceprint coding method, voiceprint coding network training method and electronic equipment

Abstract

The invention provides a voiceprint coding method, a voiceprint coding network training method and electronic equipment, and relates to the technical field of natural language processing, and the voiceprint coding method comprises the steps: obtaining to-be-processed audio data and target mask data corresponding to the to-be-processed audio data; the target mask data is used for representing an application intention of the to-be-processed audio data; and inputting the to-be-processed audio data and the target mask data into a trained voiceprint coding network, so that the voiceprint coding network fuses and reasones a coding result suitable for the application intention based on the to-be-processed audio data and the target mask data. According to the method and the device, the voiceprint coding network can be helped to better learn convergence, the situation that the convergence effect of non-convergence or different application intentions is some good and some bad is avoided, the same trained voiceprint coding network is adopted to generate the coding result for the to-be-processed audio data of different application intentions, the parameter scale can be reduced, and storage resources are saved.

Description

Technical Field

[0001] The present application relates to the technical field of natural language processing, and in particular to a voiceprint encoding method, a voiceprint encoding network training method and an electronic device. Background Art

[0002] Voiceprint encoding is the encoding of sound. Voiceprint encoding can be used to distinguish whether the voices come from the same person or different people. In order to meet the needs of security, wake-up voiceprint verification, online translation, and conference scenarios, two types of voiceprint encoding methods are needed: one is known speech content, such as voice wake-up, knowing the wake-up word in advance, and saying the wake-up word during registration and verification (content-known voiceprint). The other is unknown content, and different content is used for speech during registration and verification (content-unknown voiceprint). Therefore, two types of voiceprint encoding neural networks are required for voiceprint encoding processing, which increases network parameters and calculation amount. In addition, the above two types of voiceprint encoding usually use two independent neural networks, which are trained with different data respectively, and use two models for reasoning when used. In smart glasses or resource-constrained scenarios, it is actually a redundancy for storage and computing. In this regard, no better solution has been proposed yet. Summary of the invention

[0003] The embodiments of the present application provide a voiceprint encoding method, a voiceprint encoding network training method and an electronic device to solve one or more of the above-mentioned technical problems.

[0004] In a first aspect, an embodiment of the present application provides a voiceprint encoding method, comprising: obtaining audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to characterize the application intent of the audio data to be processed; inputting the audio data to be processed and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network infers an encoding result suitable for the application intent based on the fusion of the audio data to be processed and the target mask data.

[0005] In a second aspect, an embodiment of the present application provides a voiceprint coding network training method, wherein the voiceprint coding network includes an audio feature extraction structure, a marking structure and a main network structure; the method includes: obtaining audio training data and mask data corresponding to the audio training data; wherein the mask data is used to characterize the application intention of the audio training data; using the audio feature extraction structure, performing feature extraction on the audio training data to obtain audio feature data; associating the audio feature data and the mask data and inputting them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain voiceprint coding; adjusting the voiceprint coding network based on the voiceprint coding and a predetermined loss function to obtain a trained voiceprint coding network.

[0006] In a third aspect, an embodiment of the present application provides an electronic device, including a memory, a processor, and a computer program stored in the memory, wherein the processor implements any of the above methods when executing the computer program.

[0007] In a fourth aspect, an embodiment of the present application provides a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in any one of the above is implemented.

[0008] In a fifth aspect, an embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, it implements any of the methods described above.

[0009] Compared with the related art, this application has the following advantages:

[0010] The embodiment of the present application provides a voiceprint encoding method, a voiceprint encoding network training method and an electronic device, the voiceprint encoding method comprising: obtaining audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to characterize the application intent of the audio data to be processed; inputting the audio data to be processed and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network can infer an encoding result suitable for the application intent based on the fusion of the audio data to be processed and the target mask data. According to the embodiment of the present application, the corresponding audio data to be processed is marked with target mask data, so as to distinguish the audio data to be processed with different application intents, which can help the voiceprint encoding network to better learn and converge, and avoid the situation where there is no convergence or the convergence effects of different application intents are better or worse; the embodiment of the present application uses the same trained voiceprint encoding network to generate encoding results for the audio data to be processed with different application intents, which can reduce the parameter scale and save storage resources.

[0011] The above description is only an overview of the technical solution of the present application. In order to more clearly understand the technical means of the present application, it can be implemented in accordance with the contents of the specification. In order to make the above and other purposes, features and advantages of the present application more obvious and easy to understand, the specific implementation methods of the present application are listed below. BRIEF DESCRIPTION OF THE DRAWINGS

[0012] In the accompanying drawings, unless otherwise specified, the same reference numerals throughout the multiple drawings represent the same or similar parts or elements. These drawings are not necessarily drawn to scale. It should be understood that these drawings only depict some embodiments according to the present application and should not be regarded as limiting the scope of the present application.

[0013] Figure 1 A flow chart of a voiceprint encoding method provided in an embodiment of the present application is shown;

[0014] Figure 2 A structural diagram of a voiceprint encoding neural network provided in an embodiment of the present application is shown;

[0015] Figure 3 A network diagram of a TDNN provided in an embodiment of the present application is shown;

[0016] Figure 4 A network diagram of a D-TDNN provided in an embodiment of the present application is shown;

[0017] Figure 5 A network diagram of CAM++ provided in an embodiment of the present application is shown;

[0018] Figure 6A possible voiceprint encoding network structure diagram provided in an embodiment of the present application is shown;

[0019] Figure 7 A network structure diagram of CAM++ provided in an embodiment of the present application is shown;

[0020] Figure 8 A structural diagram of an FCM front-end convolution module provided in an embodiment of the present application is shown;

[0021] Fig. 9 A schematic diagram of a data fusion process provided in an embodiment of the present application is shown;

[0022] Fig.10 A schematic diagram of an optimized training network structure provided in an embodiment of the present application is shown;

[0023] Fig.11 A schematic diagram of a convolution structure with residual connection provided in an embodiment of the present application is shown;

[0024] Fig.12 A network diagram of a mask-based CAM++ provided in an embodiment of the present application is shown;

[0025] Fig.13 A flow chart of a voiceprint coding network training method provided in an embodiment of the present application is shown;

[0026] Fig.14 The following is a calculation step of FBANK provided in an embodiment of the present application;

[0027] Fig.15 A block diagram of an electronic device used to implement an embodiment of the present application is shown. DETAILED DESCRIPTION

[0028] In the following, only some exemplary embodiments are briefly described. As those skilled in the art will appreciate, the described embodiments may be modified in various ways without departing from the concept or scope of the present application. Therefore, the drawings and descriptions are considered to be exemplary in nature and not restrictive.

[0029] To facilitate understanding of the technical solutions of the embodiments of the present application, the following describes the related technologies of the embodiments of the present application. The following related technologies can be combined with the technical solutions of the embodiments of the present application as optional solutions, and they all belong to the protection scope of the embodiments of the present application.

[0030] First, the voiceprint encoding network that can be used in this application is described.

[0031] The voiceprint encoding method based on neural network has gone through x-vector (a voiceprint encoding neural network), TDNN (Time Delay Neural Network), D-TDNN (Dilated Time Delay Neural Network), CAM (Context Aware Masking), and the latest is CAM++.

[0032] Figure 2 shows a structural diagram of a voiceprint encoding neural network, Figure 2 The information of Layer, Layercontext, Total context and Input×output is shown, which is used to represent the network structure of the typical voiceprint coding network x-vector. X-vector is the earliest voiceprint coding neural network. The first five layers are TDNN. The meaning of time-delay neural network is to use multi-layer NN (Neural Network) to expand the time dimension vision, that is, to include audio information for a longer time at the same time, and to refer to more and more complete audio information for reasoning. Figure 3 This is a schematic diagram of the network structure of TDNN. Figure 3 The network structure of a four-layer TDNN is shown. Each layer uses a rectangular box to represent the input and output frames. The network input is from t-13 to t+9, and the output is t. For example, the first layer, layer 1, requires 5 frames of data for each calculation, the previous 2 frames, the next 2 frames, and the current frame. The output of the first layer is used as the input of the second layer. The second layer, layer 2, inputs 4 frames each time, the previous 1 frame, the next 2 frames, and the current frame. And so on until the output of layer 4.

[0033] Later, D-TDNN appeared, which took into account earlier information based on TDNN and achieved better results. Figure 4 In the above figure, there is a Feed Forward Neural Network (FNN) before TDNN, and the output of the feed forward neural network is concatenated with the result of TDNN as the input of the next layer.

[0034] The latest voiceprint encoding neural network is CAM++, as follows Figure 5 As shown, it includes FCM (pre-convolution module) and multiple DenseBlock layers. FCM is composed of ResBlock (residual block), and DenseBlock is mainly composed of TDNN plus CAM. Figure 5"Acoustic feature" in the figure indicates the input audio feature, "ReLU" indicates the activation function, "Global Pooling" indicates global pooling, and "Segment Pooling" indicates segment pooling. "Sigmoid" indicates the Sigmoid function, which is a commonly used activation function. Figure 5 The first half of the network is FCM (Front Convolution Modules), which converts the frequency domain features of a continuous audio segment into a feature vector for subsequent use. There are usually 4 residual connection convolutions (ResBlock-1 / 2 / 3 / 4 in the figure). Figure 5 The second half of the network is D-TDNN (Densely connected time delay neural network), and the number of Dense Block layers is variable.

[0035] Voiceprint coding is the coding (vector) of sound, and the similarity of coding is also the similarity of vector. The voiceprint coding of each person's speech is different. The similarity of voiceprint coding corresponding to each person's multiple speeches is relatively large, and the similarity of speech of different people is relatively small. Based on this principle, sound coding (voiceprint) can be used to distinguish whether the voice comes from the same person or different people.

[0036] Smart devices with voice assistants, such as mobile phones, smart glasses, smart watches, smart speakers, etc., basically support voice assistants and voice wake-up. Due to security issues, voice wake-up needs to support voiceprint verification. Only the owner's own voice can wake up his own device to prevent the device from being woken up by non-owners. Wake-up voiceprint verification compares the wake-up audio spoken by the user with the registered wake-up audio. If they are similar and the similarity is greater than the preset threshold, it is considered to be the owner's wake-up command, and the device wakes up. Otherwise, this wake-up is ignored. In translation scenarios and conference scenarios, it is hoped to identify the current speaker, and even separate the speech content of a specific speaker from the mixed audio.

[0037] In order to meet the above multiple applications, two types of voiceprint encoding methods are needed: one is known speech content, such as voice wake-up, the wake-up word is known in advance, and the wake-up word is spoken during registration and verification (content-known voiceprint). The other is unknown content, and different speech content is used during registration and verification. For example, in translation scenarios and conference scenarios, the speech content that needs to be separated from the mixed audio is unknown. In this way, two types of voiceprint encoding neural networks need to be trained to encode voiceprints with known content and voiceprints with unknown content respectively, which increases network parameters and computational complexity.

[0038] The above two types of voiceprint encoding usually use two independent neural networks, which are trained with different data and inferred by two models respectively. In smart glasses or resource-constrained scenarios, this is actually a redundancy of storage and computing, but there is currently no good solution.

[0039] Based on this, the present application provides a voiceprint encoding method, a voiceprint encoding network training method and an electronic device. The voiceprint encoding method uses the same neural network and is expanded to realize two task functions. It can not only infer the voiceprint encoding of audio with known content, but also infer the voiceprint encoding of audio with unknown content, thereby reducing the parameter scale and saving storage resources. The same neural network is forced to fit two tasks, that is, to learn two types of data, which may not converge, or one task is good and the other is not. In order to better fit, the method can add a mask to distinguish whether it is a wake-up voiceprint (voiceprint with known content) or a voiceprint with unknown content, to help the network learn and converge better. It can even support multiple voiceprints with known content and voiceprints with unknown content at the same time, as long as the length of the mask is extended.

[0040] The technical solution of the present application and how the technical solution of the present application solves the above-mentioned technical problems are described in detail below with specific embodiments. The several specific embodiments listed can be combined with each other, and the same or similar concepts or processes may not be repeated in some embodiments. The embodiments of the present application will be described in detail below with reference to the accompanying drawings.

[0041] The embodiment of the present application provides a voiceprint encoding method, which can be applied to smart wearable devices, such as smart glasses. Figure 1 The flowchart of the voiceprint encoding method according to an embodiment of the present application is shown. The method may include:

[0042] Step S101, obtaining audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to represent the application intention of the audio data to be processed.

[0043] In an embodiment of the present application, the audio data to be processed is audio or frequency domain feature data generated based on the user's natural language, and may be in the following forms, including but not limited to: FBANK (Filter Bank, a sound feature extraction method commonly used in speech recognition and audio signal processing) or MFCC (Mel Frequency Cepstral Coefficients). Among them, MFCC is a feature extraction method widely used in speech recognition, speaker recognition and other audio signal processing fields. MFCC aims to imitate the characteristics of the human auditory system, especially in the extraction of spectral features of speech signals. It can effectively capture the sound quality information of speech while reducing the data dimension.

[0044] It should be noted that FBANK is obtained by summing the squares of the power spectrum of the Mel filter and then taking the logarithm. Compared with MFCC, it omits the calculation of the discrete cosine transform. The calculation steps of FBANK are as follows: Fig.14 As shown. Generally, the audio data is pre-emphasized first to enhance the high-frequency signal; then it is framed and windowed. Framing is to separate the audio signal into 10ms frames. Windowing is to prevent spectrum leakage. Each time, 25ms of signal is used to calculate the feature, that is, each time it moves 10ms, and 25ms of signal is actually used, with 15ms of historical overlapping information. Then, the time domain signal is used to obtain the frequency domain signal, that is, the spectrum, through Fourier transform. The frequency domain of a certain time is accumulated to obtain the spectrum, and then the frequency is mapped to the Mel frequency scale through the Mel filter bank, and finally the logarithm is taken to obtain the FBANK feature. In the application, the Mel filter bank is generally 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0045] The target mask data is an input feature of the voiceprint encoding network, which is used to characterize the application intent of the audio data to be processed. Application intent refers to the goal or purpose that the user hopes to achieve when interacting with an application or system. For example, the application intent includes that the current audio data to be processed is used to wake up the device, or it is desired to perform voiceprint separation on the current audio data to be processed. Among them, the content contained in the current audio data to be processed for waking up the device is known, for example, the known content is: brook, brook. Among them, it is desired to perform voiceprint separation on the current audio data to be processed. For example, the current audio data to be processed includes voiceprints of multiple different people, and the voiceprint of the user of the smart glasses can be separated from the multiple voiceprints. The dimension of the target mask data can be set according to actual needs, and the embodiment of the present application does not specifically limit this. For example, when the dimension is one-dimensional, the value 1 can be used to represent that the content is known, and the value 0 can represent that the content is unknown.

[0046] It should be noted that in this step, for example, if you need to support multiple application intents, you only need to expand the dimension of the mask, for example, 2 dimensions, to represent 4 numbers, 3 of which represent different known contents and 1 represents unknown content.

[0047] Table 1 shows the voiceprint encoding of three wake-up words: Xiaoxi Xiaoxi, Xiaoji Xiaoji, and Hao Nijili.

[0048] Mask category 00 Current network infers voiceprint encoding of unknown content 01 The current network infers the voiceprint encoding of the known content 1, such as "Xiaoxi Xiaoxi" 10 The current network infers the voiceprint encoding of known content 2, such as "Xiao Ji Xiao Ji" 11 The current network infers the voiceprint encoding of known content 1, such as "Hello, Jili"

[0049] Table 1

[0050] In this step, the corresponding audio data to be processed is marked with the target mask data, so as to distinguish the audio data to be processed with different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid the situation where it does not converge or the convergence effects of different application intentions are better or worse.

[0051] Step S102: input the audio data to be processed and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network can infer an encoding result suitable for the application intention based on the fusion of the audio data to be processed and the target mask data.

[0052] In the embodiment of the present application, speech voiceprint encoding is to encode a segment of audio into a vector. The similarity of voiceprint vectors encoded from audios of different people is relatively small, that is, the voiceprints are different. The similarity of voiceprint encoding obtained from multiple audios of each person is relatively high, that is, the voiceprints are relatively close. The encoding result is a high-dimensional vector obtained by performing speech voiceprint encoding processing on the audio data to be processed and the target mask data, such as 64 / 128 / 256 dimensions.

[0053] The voiceprint encoding network can be a neural network, for example, including but not limited to: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), BERT (Bidirectional Encoder Representations from Transformers), and CAM+ (a voiceprint encoding network), etc. The audio features are passed through a certain neural network to obtain a multi-dimensional vector, such as a 32 / 64 / 128-dimensional vector. This vector is the voiceprint corresponding to the input audio and also represents the user's voice features. Figure 6 A possible voiceprint coding network structure diagram is shown, and the voiceprint coding network includes a structure for audio feature extraction and a structure for voiceprint coding, which may specifically be an X-vector, TDNN or D-TDNN. Figure 7A network structure diagram of CAM++ is shown, which specifically includes a structure for audio feature extraction, FCM and D-TDNN, wherein FCM includes multiple ResBlocks and D-TDNN includes multiple DenseBlocks.

[0054] In this step, the same trained voiceprint coding network is used to generate coding results for the audio data to be processed with different application intentions, which can reduce the parameter scale and save storage resources.

[0055] The embodiment of the present application provides a voiceprint encoding method, which includes: obtaining audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to characterize the application intent of the audio data to be processed; inputting the audio data to be processed and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network can infer an encoding result suitable for the application intent based on the fusion of the audio data to be processed and the target mask data. According to the embodiment of the present application, the corresponding audio data to be processed is marked with target mask data, so as to distinguish the audio data to be processed with different application intents, which can help the voiceprint encoding network to better learn and converge, and avoid non-convergence or the convergence effect of different application intents is better or worse; the embodiment of the present application uses the same trained voiceprint encoding network to generate encoding results for the audio data to be processed with different application intents, which can reduce the parameter scale and save storage resources.

[0056] In a possible implementation, obtaining the audio data to be processed and the target mask data corresponding to the audio data to be processed can be performed according to the following steps: obtaining the audio data to be processed in a non-awakened state; performing voice wake-up recognition on the audio data to be processed; in response to the recognition result of the voice wake-up recognition being wake-up, determining that the target mask data of the audio data to be processed is the mask data corresponding to the wake-up audio; the method also includes: obtaining a registered voiceprint feature; verifying the encoding result of the voiceprint encoding according to the registered voiceprint feature, and determining whether to execute the wake-up.

[0057] In this possible implementation, the non-awakened state refers to the system, device or model being in a low-power or dormant state, waiting for a specific trigger event to enter a working state. In this state, the system consumes minimal resources, and the main purpose is to save energy and resources. The audio data to be processed is obtained in the non-awakened state, and voice wake-up recognition is performed on the audio data to be processed. Voice wake-up recognition is a speech recognition technology used to detect specific trigger words or phrases to wake up the device or start a specific function. If the audio data to be processed includes a specific trigger word or phrase, the recognition result of the voice wake-up recognition is determined to be wake-up, and the target mask data of the audio data to be processed is determined to be the mask data corresponding to the wake-up audio, that is, the correspondence between the two is determined, wherein the wake-up audio can be used as data with known content. Based on this step, after the wake-up recognition is successful, the voice frame used for wake-up recognition can be added with the mask corresponding to the voiceprint recognition and then input into the voiceprint encoding model.

[0058] In this possible implementation, a registered voiceprint feature may also be obtained. The registered voiceprint feature is obtained by collecting and processing the user's voice sample, extracting and storing a unique voiceprint feature for subsequent identity verification or identification. The encoding result of the voiceprint encoding is verified according to the registered voiceprint feature to determine whether to perform the wake-up. For example, the registered voiceprint feature and the encoding result of the voiceprint encoding may be compared. When the similarity between the two exceeds a preset similarity threshold, it is determined to perform the wake-up, thereby ensuring the security of the wake-up operation.

[0059] In one possible implementation, multiple wake-up words may be supported, as shown in Table 1. In this case, the wake-up word included in the audio data to be processed may be determined by voice wake-up recognition. Then, the mask data corresponding to the wake-up word included in the audio data to be processed may be used as the target mask data.

[0060] The following examples illustrate the steps of voice wake-up voiceprint verification: 1) User registration wake-up voiceprint: The user can say the wake-up word multiple times, obtain multiple voiceprints, and take the average. Or the registered audio contains a long audio with multiple wake-up words, and a voiceprint is calculated. The voiceprint obtained in this step is the registered voiceprint. 2) User wake-up: When in use, the user says the wake-up word to wake up. 3) The wake-up audio calculates the voiceprint. If the voice wakes up, the user wake-up audio is processed by FBANK or MFCC method for sound feature extraction to obtain the audio data to be processed, and then the voiceprint is calculated based on the audio data to be processed to obtain the wake-up voiceprint. 4) Calculate the similarity between the wake-up voiceprint and the registered voiceprint. 5) Similarity comparison determines whether the wake-up is passed or not. Compare the similarity calculated above with the preset threshold value. If the threshold value is exceeded, the voiceprint verification is passed, otherwise it fails.

[0061] For voiceprint authentication (verification) scenarios with the same content, it is possible to limit the voice assistant to only authenticated users who can wake up their devices. In similar scenarios, voiceprint verification requires the same speech content during registration and application, that is, the audio content spoken by the user during registration and training are the same, while the accuracy of other content cannot be guaranteed. Similar to the above-mentioned voiceprint authentication (verification) scenarios with the same content, in the voice wake-up scenario, the wake-up word is known in advance. The advantage of this is that the verification can be more rigorous, the wake-up voiceprints of registered users are closer, and the distance between the wake-up word voiceprint encoding vectors between unregistered users and registered users is larger, that is, the discrimination is greater and easier to distinguish. Therefore, voiceprint application scenarios with high security requirements can be implemented using the embodiments of the present application.

[0062] In a possible implementation, obtaining the audio data to be processed and the target mask data corresponding to the audio data to be processed can be performed according to the following steps: obtaining the audio data to be processed in an awake state; determining that the mask data corresponding to the current application scenario is the target mask data corresponding to the audio data to be processed; the method also includes: identifying and / or separating the target audio from the mixed audio data according to the encoding result of the voiceprint encoding; wherein the target audio has the same voiceprint feature as the audio data to be processed.

[0063] In the embodiment of the present application, the awake state refers to the system, device or model being in an active working state, capable of processing tasks and performing operations. In this state, the system consumes more resources, but can quickly respond to user requests or process data. The audio data to be processed is obtained in the awake state. In the awake state, the execution subject of the method may be applied to a variety of different application scenarios. Different mask data are pre-set in different application scenarios. The mask data corresponding to the current application scenario of the execution subject is determined to be the target mask data corresponding to the audio data to be processed. In this step, corresponding masks are set for different application scenarios, and the glasses actively select the corresponding mask according to the application scenario.

[0064] In this possible implementation, the target audio can also be identified and / or separated from the mixed audio data based on the encoding result of the voiceprint encoding, wherein the mixed audio data may include multiple audios with different voiceprint characteristics. After obtaining the encoding result of the voiceprint encoding, the target audio having the same voiceprint characteristics as the audio data to be processed can be identified and / or separated.

[0065] The following example illustrates the steps of voiceprint-based speech separation: the user registers an audio voiceprint and obtains his own voiceprint, which has nothing to do with the content of the speech. In a multi-person speech scene, for example, a person wearing glasses and talking to the person opposite, the microphone is on the glasses, and the recording contains the voice of the glasses wearer and the person opposite. The speech of the glasses wearer and the person opposite are separated from the mixed audio by a speech separation neural network. The speech separation neural network uses the voiceprint of the glasses wearer as a feature to improve the separation effect.

[0066] When the device or apparatus using this method uses a voice assistant, in a voiceprint encoding scenario where the content is unknown, the user's speech content can be arbitrary, and the voiceprint encoding of this audio is performed to obtain a vector, and then the similarity between multiple voiceprint encodings obtained from multiple speech audios is compared, and a similarity threshold is preset. If the threshold is exceeded, it is considered to be the same person, otherwise it is different people. This voiceprint encoding usage mainly involves the following two types: one is to distinguish who is the current speaker in a multi-person speech scenario; the other is an application based on voiceprint separation, in which the voiceprint encoding can be used as part of the input of the speech separation network.

[0067] In a possible implementation, the voiceprint encoding network includes an audio feature extraction structure, a marking structure and a main network structure, and the audio data to be processed and the target mask data are input into a trained voiceprint encoding network so that the voiceprint encoding network can infer an encoding result suitable for the application intention based on the fusion of the audio data to be processed and the target mask data, including: using the audio feature extraction structure to extract features from the audio data to be processed to obtain audio feature data; associating the audio feature data and the mask data into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature with time and frequency characteristics; and inputting the marked audio feature data into the main network to obtain the voiceprint encoding.

[0068] In this possible implementation, when the audio data to be processed is speech data, for example, a recording of a speech spoken by a user, the audio feature extraction structure can obtain audio feature data, such as FBANK and MFCC, based on the audio data to be processed. The following is an explanation of the process of obtaining FBANK as an example.

[0069] Input: A segment of audio, collected continuously as input. The audio sampling rate is generally 16kHz, the bit width is 16bit, and it is single channel. Output: Feature value, for example, if this segment of audio is 160ms, it is divided into 16 frames, and the feature of each frame is 40-dimensional data, then the 16-frame feature is [16,40]. Function: In order to process real-time, the audio is cut into segments, usually 10ms as a frame. Generally speaking, a segment of 10ms of audio can extract meaningful features and results. In the subsequent processing process, in order to prevent spectrum leakage, the method of partially overlapping the previous and next frames is adopted during processing. The frame length is 25ms each time, of which 15ms is historical information, that is, 15ms overlap, and the actual movement is only 10ms.

[0070] In the embodiment of the present application, the marking structure includes the FCM front-end convolution module, see Figure 8 The FCM front-end convolution module structure shown in the figure can be composed of multiple layers of ResBlock, including ResBlock1-4, and each ResBlock is composed of convolution and residual connection. Fig.11 A typical convolutional structure with residual connections is shown. Fig.11 Shown is Figure 8 The association relationship between the convolutions in each ResBlock in , where add is to add the data of two branches together. After FCM processing, multi-frame audio feature data (FBANK or MFCC) is converted into features with time and frequency characteristics. That is, the audio feature data and the mask data are associated and input into the tag structure to obtain the labeled audio feature data, and the labeled audio feature data is a feature with time and frequency characteristics.

[0071] The tag structure may also include a fusion module, which is used to combine multiple features together. There are many ways to combine, including one or more of the following processes: vector concatenation, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication, and linear combination. For example, features A and B are one-dimensional vectors, and they can be fused together by any of the following fusion methods: dimension expansion fusion: [A, B]; addition fusion: A+B, provided that the dimensions of A and B are consistent; multiplication fusion: A*B. In the embodiment of the present application, dimension expansion fusion can be preferably used.

[0072] Finally, the marked audio feature data is input into the main network to obtain the voiceprint code. The network structure of the main network can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. Preferably, the main network is a DenseBlock network including a TDNN (time delay neural network). Figure 5As shown in the figure, a CAM++ network diagram can be selected as the main network. Figure 5 The main network contains 3 layers of DenseBlock, each layer of DenseBlock consists of FNN (Feedforward Neural Network), TDNN and CAM (Context Aware Masking). Among them, FNN is a general neural network, such as a convolutional layer or a fully connected layer, consisting of multiple layers. TDNN is a time-delay neural network, consisting of multiple fully connected layers, each layer is staggered in the time dimension, so as to extract a large amount of information in the time dimension. The Context in CAM indicates whether the current frame is the target speaker or noise interference. See Fig.12 The network diagram of mask-based CAM++ is shown in Fig.12 The M in is a scalar, which can represent the probability that the current frame is the target speaker's speech. For example, close to 1 means that the current frame is the target speech audio, and close to 0 means that the current frame is interference. M acts as a weight on the result of TDNN. Each layer of TDNN has a CAM. The output of FNN is used as the input of CAM, and M is the output result of CAM in the figure. The value range of M is 0 to 1. If the current frame is interference data, the interference data will be filtered out when M is 0. CAM is used to filter out interference data in the input data of TDNN.

[0073] In a possible implementation, the labeling structure includes a fusion module and a front-end convolution module, and the audio feature data and the mask data are associated and input into the labeling structure to obtain the labeled audio feature data, including: inputting the audio feature data and the mask data into the fusion module, and using the fusion module to fuse the mask data into the audio feature data; inputting the audio feature data with the mask data added into the front-end convolution module, and using the front-end convolution module to convert the audio feature data fused with the mask data into the labeled audio feature data.

[0074] In this possible implementation, see Fig. 9 In the "Multi-task voiceprint encoding network 1" shown in the figure, the mask input information can be fused with the audio features as the input of the FCM, that is, the audio feature data and the mask data are input into the fusion module, and the mask data is fused into the audio feature data by using the fusion module. After that, the audio feature data with the mask data added is input into the front-end convolution module, and the audio feature data fused with the mask data is converted into the marked audio feature data by using the front-end convolution module.

[0075] It should be noted that the structure of the voiceprint encoding network can be Fig. 9 The network shown in the figure is based on full connection and Softmax. The function of full connection is to map the dimension to the number of categories. For example, if there are 4 categories, the full connection output is 4 dimensions. The function of Softmax is to perform nonlinear transformation on the values ​​of each category, making the training faster and more effective. The optimized network structure is as follows Fig.10 shown.

[0076] In a possible implementation, the labeling structure includes a fusion module and a front-end convolution module, and the audio feature data and the mask data are associated and input into the labeling structure to obtain the labeled audio feature data, including: inputting the audio feature data into the front-end convolution module, and using the front-end convolution module to convert the audio feature data into a feature extraction result with time and frequency characteristics; inputting the feature extraction result and the mask data into the fusion module, and using the fusion module to fuse the mask data into the feature extraction result to obtain the labeled audio feature data.

[0077] In this possible implementation, see Fig. 9 In the "Multi-task Voiceprint Coding Network 2" shown in the figure, the information of the mask input can be combined with the output of the FCM as the input of the DenseBlock layer, that is, the audio feature data is input into the front-end convolution module, and the front-end convolution module is used to convert the audio feature data into a feature extraction result with time and frequency characteristics. When implementing this step, for example, the input is an audio feature frame feature (FBANK or MFCC) within a period of time, usually one frame every 25ms. Assuming that 2 seconds of data is used as input, the 2-second audio has 80 frames of FBANK features. Assuming that the FBANK feature is 40-dimensional, the 2-second data is an array of [80,40]. The output is a feature containing time characteristics and frequency domain characteristics.

[0078] Afterwards, the feature extraction result and the mask data are input into the fusion module, and the mask data is fused into the feature extraction result by the fusion module to obtain the marked audio feature data. In this possible implementation, the amount of calculation and the scale of parameters are small.

[0079] The embodiment of the present application also provides a voiceprint coding network training method, wherein the voiceprint coding network includes an audio feature extraction structure, a marking structure and a main network structure; Fig.13 The flowchart of the voiceprint coding network training method according to an embodiment of the present application is shown. The method may include:

[0080] Step S1301: Acquire audio training data and mask data corresponding to the audio training data; wherein the mask data is used to represent the application intention of the audio training data.

[0081] In an embodiment of the present application, the audio training data is audio data. The mask data is an input feature of the voiceprint encoding network, which is used to characterize the application intent of the audio training data. Application intent refers to the goal or purpose that the user hopes to achieve when interacting with an application or system. For example, the application intent includes whether the content contained in the current audio data to be processed is known or unknown. The dimension of the mask data can be set according to actual needs, and the embodiment of the present application does not specifically limit this. For example, when the dimension is one-dimensional, the value 1 can be used to represent that the content is known, and the value 0 can represent that the content is unknown.

[0082] It should be noted that in this step, for example, if you need to support multiple known contents, you only need to expand the dimension of the mask, for example, 2 dimensions, to represent 4 numbers, 3 of which represent different known contents and 1 represents unknown content. As shown in Table 1, three types of voiceprint encoding of wake-up words are supported: Xiaoxi Xiaoxi, Xiaoji Xiaoji, and Hao Hao Jili.

[0083] The implementation of this step is described below with a specific embodiment. The audio training data is divided into two types, one with known content and the other with unknown content. Because the mask input is added, the mask of the known content corresponds to the type of known content, and the mask of the unknown content is 0. Taking the three types of voiceprint encoding with known content as an example, the data is organized into the format shown in Table 2 below, where the third column is used to characterize whether the input audio content belongs to the unknown content category or the known content category:

[0084]

[0085] Table 2

[0086] Step S1302: Using the audio feature extraction structure, extract features from the audio training data to obtain audio feature data.

[0087] In a possible implementation, feature extraction is performed on the audio training data to obtain audio feature data, including: generating FBANK data and / or MFCC data based on the audio training data; and using the FBANK data and / or the MFCC data as audio feature data.

[0088] In an embodiment of the present application, the audio feature data is frequency domain feature data generated based on the natural language of the user, and may be in the form of, but not limited to, FBANK (Filter Bank, a sound feature extraction method commonly used in speech recognition and audio signal processing) or MFCC (Mel Frequency Cepstral Coefficients). Among them, MFCC is a feature extraction method widely used in speech recognition, speaker recognition, and other audio signal processing fields. MFCC is intended to mimic the characteristics of the human auditory system, especially in the extraction of spectral features of speech signals. It can effectively capture the sound quality information of speech while reducing the data dimension.

[0089] It should be noted that FBANK is obtained by summing the squares of the power spectrum of the Mel filter and then taking the logarithm. Compared with MFCC, it omits the calculation of the discrete cosine transform. The calculation steps of FBANK are as follows: Fig.14 As shown. Generally, the audio data is pre-emphasized first to enhance the high-frequency signal; then it is framed and windowed. Framing is to separate the audio signal into 10ms frames. Windowing is to prevent spectrum leakage. Each time, 25ms of signal is used to calculate the feature, that is, each time it moves 10ms, and 25ms of signal is actually used, with 15ms of historical overlapping information. Then, the time domain signal is used to obtain the frequency domain signal, that is, the spectrum, through Fourier transform. The frequency domain of a certain time is accumulated to obtain the spectrum, and then the frequency is mapped to the Mel frequency scale through the Mel filter bank, and finally the logarithm is taken to obtain the FBANK feature. In the application, the Mel filter bank is generally 40 or 80, that is, each frame of audio corresponds to 40 or 80 outputs.

[0090] Step S1303: associate the audio feature data and the mask data and input them into the tag structure to obtain the tagged audio feature data, where the tagged audio feature data has features of time and frequency characteristics.

[0091] In this step, the mask data is used to mark the corresponding audio feature data, so as to distinguish the audio feature data of different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid non-convergence or different application intentions The convergence effect is better or worse.

[0092] In a possible implementation, the labeling structure includes a fusion module and a front-end convolution module, the output end of the fusion module is connected to the input end of the front-end convolution module, and the audio feature data and the mask data are associated and input into the labeling structure to obtain the labeled audio feature data, including: inputting the audio feature data and the mask data into the fusion module, and using the fusion module to add the mask data to the audio feature data; inputting the audio feature data with the mask data added into the front-end convolution module, and using the front-end convolution module to extract the time domain data and frequency domain data of the audio feature data with the mask data added to obtain the labeled audio feature data. Preferably, adding the mask data to the audio feature data includes: performing dimension expansion and fusion on the audio feature data and the mask data; wherein the dimension expansion and fusion includes one or more of the following processing: vector concatenation, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication, and linear combination.

[0093] In a possible implementation, the labeling structure includes a fusion module and a front-end convolution module, the output end of the front-end convolution module is connected to the input end of the fusion module, and the audio feature data and the mask data are associated and input into the labeling structure to obtain the labeled audio feature data, including: inputting the audio feature data into the front-end convolution module, and using the front-end convolution module to extract the time domain data and frequency domain data of the audio feature data to obtain the extraction result; inputting the extraction result and the mask data into the fusion module, and using the fusion module to add the mask data to the extraction result. Preferably, adding the mask data to the extraction result includes: performing dimension expansion and fusion on the extraction result and the mask data; wherein the dimension expansion and fusion includes one or more of the following processing: vector concatenation, vector element-by-element multiplication, vector element-by-element addition, matrix multiplication, and linear combination.

[0094] Step S1304: input the marked audio feature data into the main network to obtain the voiceprint code.

[0095] Speech voiceprint encoding is to encode a piece of audio into a vector. The similarity of voiceprint vectors encoded from audios of different people is relatively small, that is, the voiceprints are different. The similarity of voiceprint encodings obtained from multiple audios of each person is relatively high, that is, the voiceprints are relatively close. The voiceprint encoding output by the main network is a high-dimensional vector obtained by performing speech voiceprint encoding processing on the processed audio data and the target mask data, such as 64 / 128 / 256 dimensions.

[0096] The voiceprint encoding network can be a neural network, for example, including but not limited to: CNN (Convolutional Neural Network), LSTM (Long Short-Term Memory), BERT (Bidirectional Encoder Representations from Transformers), and CAM+ (Class Activation Mapping++, a voiceprint encoding neural network), etc. The audio features are passed through a certain neural network to obtain a multi-dimensional vector, such as a 32 / 64 / 128-dimensional vector. This vector is the voiceprint corresponding to the input audio and also represents the user's voice features. Figure 6 A possible voiceprint encoding network structure diagram is shown. Figure 7 A network structure diagram of CAM++ is shown.

[0097] In one possible implementation, the main network structure includes multiple densely connected blocks; the densely connected blocks include: a feedforward neural network, a time-delay neural network and a CAM module; wherein the CAM module is used to filter out interference data in the input data of the time-delay neural network module.

[0098] In the embodiment of the present application, the network structure of the main network can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. Preferably, the main network is a DenseBlock network including a TDNN (time delay neural network). Figure 5 As shown in the figure, a CAM++ network diagram can be selected as the main network. Figure 5 The main network contains 3 layers of DenseBlock, each layer of DenseBlock consists of FNN (Feedforward Neural Network), TDNN and CAM (Context Aware Masking). Among them, FNN is a general neural network, such as a convolutional layer or a fully connected layer, consisting of multiple layers. TDNN is a time-delay neural network, consisting of multiple fully connected layers, each layer is staggered in the time dimension, so as to extract a large amount of information in the time dimension. The Context in CAM indicates whether the current frame is the target speaker or noise interference. See Fig.12 The network diagram of mask-based CAM++ is shown in Fig.12The M in is a scalar, which can represent the probability that the current frame is the target speaker's speech. For example, close to 1 means that the current frame is the target speech audio, and close to 0 means that the current frame is interference. M acts as a weight on the result of TDNN. Each layer of TDNN has a CAM module. The output of FNN is used as the input of CAM, and M is the output result of the CAM module in the figure. The value range of M is 0 to 1. The CAM module is used to filter out interference data in the input data of TDNN.

[0099] In this step, the same trained voiceprint coding network is used to generate voiceprint coding for audio training data of different application intentions, which can reduce the parameter scale and save storage resources.

[0100] Step S1405: Based on the voiceprint coding and a predetermined loss function, the voiceprint coding network is adjusted to obtain a trained voiceprint coding network.

[0101] In the embodiment of the present application, the loss function can be selected according to actual needs, and the embodiment of the present application does not specifically limit this. For example, the predetermined loss function may include but is not limited to the following types: loss functions based on EuclidDistance: Contrastive Loss, Triplet Loss, CenterLoss, etc.; loss functions related to Angular Margin: L-Softmax Loss (Large MarginSoftmax Loss, large interval soft maximum loss), A-Softmax Loss (Angular Softmax Loss, angle soft maximum loss), CosFace Loss (cosine face loss), ArcFace Loss (Additive Angular Margin Loss, additive angle interval loss), etc.

[0102] The voiceprint encoding network obtained by the above loss function can achieve the effect of large inter-class vector distance and small intra-class vector distance of voiceprint encoding. Among them, the inter-class vector distance is large, that is, the distance between samples of different categories in the feature space should be as large as possible to ensure the distinction between different categories. This can reduce the confusion between samples of different categories. The intra-class vector distance is small, that is, the distance between samples of the same category in the feature space should be as small as possible to ensure that samples of the same category are clustered together. This helps to improve the similarity of samples of the same category.

[0103] An embodiment of the present application provides a voiceprint coding network training method, wherein the voiceprint coding network includes an audio feature extraction structure, a marking structure and a main network structure; the method includes: obtaining audio training data and mask data corresponding to the audio training data; wherein the mask data is used to characterize the application intention of the audio training data; using the audio feature extraction structure to extract features from the audio training data to obtain audio feature data; associating the audio feature data and the mask data and inputting them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature with time and frequency characteristics; inputting the marked audio feature data into the main network to obtain voiceprint coding; adjusting the voiceprint coding network based on the voiceprint coding and a predetermined loss function to obtain a trained voiceprint coding network. According to the embodiment of the present application, the corresponding audio training data is marked with mask data to distinguish the audio training data of different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid situations where convergence does not converge or the convergence effects of different application intentions are better or worse. The embodiment of the present application uses the same voiceprint coding network to train the audio training data of different application intentions, which can reduce the parameter scale and save storage resources.

[0104] In a possible implementation, before obtaining the mask data corresponding to the audio training data, it also includes: determining the number of categories of the content in the training set where the audio training data is located; determining the number of mask data of the mask set according to the number of categories, and generating the mask set according to the number of mask data; obtaining the mask data corresponding to the audio training data includes: determining the mask data corresponding to the audio training data in the mask set. The number of categories is used to characterize the total number of categories of content with known content and unknown content included in the training set. For example, the training set includes 5 groups of audio training data with the content of "Xiaoxi Xiaoxi", and these 5 groups of audio training data can be "Xiaoxi Xiaoxi" spoken by different people. The training set also includes 8 groups of audio training data with the content of "Xiao Ji Xiao Ji", and these 8 groups of audio training data can be "Xiao Ji Xiao Ji" spoken by different people. In addition, the training set also includes a group of audio training data with unknown content. The training set also includes a group of audio training data with the content of "Hello Ji Li", and the number of categories included in the training set is 4, that is, the categories with known content are: "Xiaoxi Xiaoxi", "Xiao Ji Xiao Ji" and "Hello Ji Li", and the other is a category with unknown content. Based on the number of categories 4, it can be determined that the number of mask data in the mask set should be at least 4, and then a two-dimensional mask set can be determined, using "00", "01", "10" and "11" to represent masks of different categories respectively.

[0105] Corresponding to the application scenario and the voiceprint encoding method provided in the embodiment of the present application, the embodiment of the present application further provides a voiceprint encoding device, which may include:

[0106] An acquisition module is used to acquire audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to characterize the application intent of the audio data to be processed; an encoding module is used to input the audio data to be processed and the target mask data into a trained voiceprint encoding network, so that the voiceprint encoding network can infer an encoding result suitable for the application intent based on the fusion of the audio data to be processed and the target mask data.

[0107] According to the embodiment of the present application, the corresponding audio data to be processed is marked with target mask data, so as to distinguish the audio data to be processed with different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid the situation where it does not converge or the convergence effects of different application intentions are better or worse. The embodiment of the present application uses the same trained voiceprint coding network to generate coding results for the audio data to be processed with different application intentions, which can reduce the parameter scale and save storage resources.

[0108] In one possible implementation, the acquisition module is specifically used to: acquire the audio data to be processed in an unawakened state; perform voice wake-up recognition on the audio data to be processed; in response to the recognition result of the voice wake-up recognition being wake-up, determine that the target mask data of the audio data to be processed is the mask data corresponding to the wake-up audio; the device also includes an execution module, which is used to: acquire a registered voiceprint feature; verify the encoding result of the voiceprint encoding according to the registered voiceprint feature, and determine whether to execute the wake-up.

[0109] In one possible implementation, the acquisition module is specifically used to: acquire the audio data to be processed in an awake state; determine that the mask data corresponding to the current application scenario is the target mask data corresponding to the audio data to be processed; the device also includes a processing module, used to: identify and / or separate the target audio from the mixed audio data according to the encoding result of the voiceprint encoding; wherein the target audio has the same voiceprint characteristics as the audio data to be processed.

[0110] In a possible implementation, the voiceprint encoding network includes an audio feature extraction structure, a marking structure and a main network structure, and the encoding module is specifically used to: use the audio feature extraction structure to extract features from the audio data to be processed to obtain audio feature data; associate the audio feature data and the mask data and input them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data has features with time and frequency characteristics; input the marked audio feature data into the main network to obtain the voiceprint code.

[0111] In a possible implementation, the marking structure includes a fusion module and a front-end convolution module, and the encoding module is specifically used to: input the audio feature data and the mask data into the fusion module, and use the fusion module to fuse the mask data into the audio feature data; input the audio feature data with the mask data added into the front-end convolution module, and use the front-end convolution module to convert the audio feature data fused with the mask data into the marked audio feature data.

[0112] In a possible implementation, the marking structure includes a fusion module and a front-end convolution module, and the encoding module is specifically used to: input the audio feature data into the front-end convolution module, and use the front-end convolution module to convert the audio feature data into a feature extraction result with time and frequency characteristics; input the feature extraction result and the mask data into the fusion module, and use the fusion module to fuse the mask data into the feature extraction result to obtain the marked audio feature data.

[0113] Corresponding to the application scenario of the voiceprint coding network training method and the voiceprint coding network training method provided in the embodiment of the present application, the embodiment of the present application also provides a voiceprint coding network training device, wherein the voiceprint coding network includes an audio feature extraction structure, a marking structure and a main network structure, and the device may include:

[0114] A data module, used to obtain audio training data and mask data corresponding to the audio training data; wherein the mask data is used to characterize the application intention of the audio training data; an extraction module, used to use the audio feature extraction structure to extract features from the audio training data to obtain audio feature data; an association module, used to associate the audio feature data and the mask data and input them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature with time and frequency characteristics; an input module, used to input the marked audio feature data into the main network to obtain voiceprint coding; a training module, used to adjust the voiceprint coding network based on the voiceprint coding and a predetermined loss function to obtain a trained voiceprint coding network.

[0115] According to the embodiment of the present application, the corresponding audio training data is marked with mask data to distinguish the audio training data of different application intentions, which can help the voiceprint coding network to better learn and converge, and avoid situations where convergence does not converge or the convergence effects of different application intentions are better or worse. The embodiment of the present application uses the same voiceprint coding network to train the audio training data of different application intentions, which can reduce the parameter scale and save storage resources.

[0116] In a possible implementation, the data module is specifically used to: determine the number of categories of the content in the training set where the audio training data is located; determine the number of mask data in the mask set according to the number of categories, and generate a mask set according to the number of mask data; and determine the mask data corresponding to the audio training data in the mask set.

[0117] The functions of each module in each device in the embodiments of the present application can be found in the corresponding description in the above method, and have corresponding beneficial effects, which will not be repeated here.

[0118] Fig.15 FIG. 1 is a block diagram of an electronic device used to implement an embodiment of the present application. Fig.15 As shown, the electronic device includes: a memory 1501 and a processor 1502. The memory 1501 stores a computer program that can be run on the processor 1502. When the processor 1502 executes the computer program, the method in the above embodiment is implemented. The number of the memory 1501 and the processor 1502 can be one or more.

[0119] The electronic device also includes:

[0120] The communication interface 1503 is used to communicate with external devices and perform data exchange transmission.

[0121] If the memory 1501, the processor 1502 and the communication interface 1503 are implemented independently, the memory 1501, the processor 1502 and the communication interface 1503 can be connected to each other through a bus and communicate with each other. The bus can be an Industry Standard Architecture (ISA) bus, a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus. The bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, Fig.15 Only one thick line is used in the diagram, but this does not mean that there is only one bus or only one type of bus.

[0122] Optionally, in a specific implementation, if the memory 1501, the processor 1502 and the communication interface 1503 are integrated on a chip, the memory 1501, the processor 1502 and the communication interface 1503 can communicate with each other through an internal interface.

[0123] An embodiment of the present application provides a computer-readable storage medium storing a computer program, which implements the method provided in the embodiment of the present application when the program is executed by a processor.

[0124] An embodiment of the present application provides a computer program product, wherein the computer program product includes a computer program, and when the computer program is executed by a processor, the method provided in the embodiment of the present application is implemented.

[0125] An embodiment of the present application also provides a chip, which includes a processor for calling and executing instructions stored in the memory from the memory, so that a communication device equipped with the chip executes the method provided by the embodiment of the present application.

[0126] An embodiment of the present application also provides a chip, including: an input interface, an output interface, a processor and a memory, wherein the input interface, the output interface, the processor and the memory are connected via an internal connection path, and the processor is used to execute the code in the memory. When the code is executed, the processor is used to execute the method provided in the embodiment of the application.

[0127] It should be understood that the processor may be a central processing unit (CPU), or other general-purpose processors, digital signal processors (DSP), application-specific integrated circuits (ASIC), field programmable gate arrays (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor may be a microprocessor or any conventional processor, etc. It is worth noting that the processor may be a processor supporting the Advanced RISC Machines (ARM) architecture.

[0128] Further, optionally, the above-mentioned memory may include a read-only memory and a random access memory. The memory may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory. Among them, the non-volatile memory may include a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM), or a flash memory. The volatile memory may include a random access memory (RAM), which is used as an external cache. By way of exemplary but not limiting description, many forms of RAM are available. For example, static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (SDRAM), double data rate synchronous dynamic random access memory (DDR SDRAM), enhanced synchronous dynamic random access memory (ESDRAM), synchronous link dynamic random access memory (SLDRAM) and direct memory bus random access memory (DR RAM).

[0129] In the above embodiments, it can be implemented in whole or in part by software, hardware, firmware or any combination thereof. When implemented using software, it can be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, the process or function according to the present application is generated in whole or in part. The computer can be a general-purpose computer, a special-purpose computer, a computer network, or other programmable device. The computer instructions can be stored in a computer-readable storage medium, or transmitted from one computer-readable storage medium to another computer-readable storage medium.

[0130] It should be noted that the information (including but not limited to user device information, user personal information, etc.) and data (including but not limited to data used for analysis, stored data, displayed data, etc.) involved in this application are all information and data authorized by the user or fully authorized by all parties, and the collection, use and processing of relevant data must comply with the relevant laws, regulations and standards of relevant countries and regions, and provide corresponding operation entrances for users to choose to authorize or refuse.

[0131] In the description of this specification, the description with reference to the terms "one embodiment", "some embodiments", "example", "specific example", or "some examples" means that the specific features, structures, materials or characteristics described in conjunction with the embodiment or example are included in at least one embodiment or example of the present application. Moreover, the specific features, structures, materials or characteristics described may be combined in any one or more embodiments or examples in a suitable manner. In addition, those skilled in the art may combine and combine different embodiments or examples described in this specification and the features of different embodiments or examples, unless they are contradictory.

[0132] In addition, the terms "first" and "second" are used for descriptive purposes only and should not be understood as indicating or implying relative importance or implicitly indicating the number of technical features indicated. Therefore, a feature defined as "first" or "second" may explicitly or implicitly include at least one of the features. In the description of this application, the meaning of "plurality" is two or more, unless otherwise clearly and specifically defined.

[0133] Any process or method described in the flow chart or otherwise described herein can be understood as a module, fragment or portion of a code representing one or more executable instructions for implementing the steps of a specific logical function or process. And the scope of the preferred embodiment of the present application includes other implementations, in which the functions may not be performed in the order shown or discussed, including in a substantially simultaneous manner or in a reverse order according to the functions involved.

[0134] The logic and / or steps described in the flowchart or otherwise described herein, for example, can be considered as an ordered list of executable instructions for implementing logical functions, which can be specifically implemented in any computer-readable medium for use by an instruction execution system, device or apparatus (such as a computer-based system, a system including a processor, or other system that can fetch instructions from an instruction execution system, device or apparatus and execute instructions), or used in combination with these instruction execution systems, devices or apparatuses.

[0135] It should be understood that the various parts of the present application can be implemented with hardware, software, firmware or a combination thereof. In the above embodiments, multiple steps or methods can be implemented with software or firmware stored in a memory and executed by a suitable instruction execution system. All or part of the steps of the above embodiment method can be completed by instructing the relevant hardware through a program, which can be stored in a computer-readable storage medium, and when the program is executed, it includes one of the steps of the method embodiment or a combination thereof.

[0136] In addition, each functional unit in each embodiment of the present application can be integrated into a processing module, or each unit can exist physically separately, or two or more units can be integrated into one module. The above-mentioned integrated module can be implemented in the form of hardware or in the form of a software functional module. If the above-mentioned integrated module is implemented in the form of a software functional module and sold or used as an independent product, it can also be stored in a computer-readable storage medium. The storage medium can be a read-only memory, a disk or an optical disk, etc.

[0137] The above is only an exemplary embodiment of the present application, but the protection scope of the present application is not limited thereto. Any technician familiar with the technical field can easily think of various changes or substitutions within the technical scope recorded in the present application, which should be included in the protection scope of the present application. Therefore, the protection scope of the present application shall be based on the protection scope of the claims.

Claims

1. A voiceprint encoding method, comprising: Acquire audio data to be processed and target mask data corresponding to the audio data to be processed; wherein the target mask data is used to characterize the application intention of the audio data to be processed; The audio data to be processed and the target mask data are input into a trained voiceprint encoding network, so that the voiceprint encoding network can infer an encoding result suitable for the application intention based on the fusion of the audio data to be processed and the target mask data.

2. The method according to claim 1, wherein: Acquiring audio data to be processed and target mask data corresponding to the audio data to be processed, including: Acquire the audio data to be processed in an unawakened state; Performing voice wake-up recognition on the audio data to be processed; In response to a recognition result of the voice wake-up recognition being wake-up, determining that the target mask data of the audio data to be processed is mask data corresponding to the wake-up audio; The method further comprises: Get the registered voiceprint features; The encoding result of the voiceprint encoding is verified according to the registered voiceprint feature to determine whether to execute the wake-up.

3. The method according to claim 1, wherein: Acquiring audio data to be processed and target mask data corresponding to the audio data to be processed, including: Acquire the audio data to be processed in the awake state; Determining that mask data corresponding to the current application scenario is target mask data corresponding to the audio data to be processed; The method further comprises: According to the encoding result of the voiceprint encoding, target audio is identified and / or separated from the mixed audio data; wherein the target audio and the audio data to be processed have the same voiceprint characteristics.

4. The method according to any one of claims 1 to 3, wherein: The voiceprint encoding network includes an audio feature extraction structure, a labeling structure and a main network structure. The audio data to be processed and the target mask data are input into the trained voiceprint encoding network, so that the voiceprint encoding network can fuse and infer the encoding result suitable for the application intention based on the audio data to be processed and the target mask data, including: Using the audio feature extraction structure, extracting features from the audio data to be processed to obtain audio feature data; Associating the audio feature data and the mask data and inputting them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature having time and frequency characteristics; The marked audio feature data is input into the main network to obtain the voiceprint code.

5. The method according to claim 4, wherein: The tagging structure includes a fusion module and a front-end convolution module, and the audio feature data and the mask data are associated and input into the tagging structure to obtain the tagged audio feature data, including: Inputting the audio feature data and the mask data into the fusion module, and using the fusion module to fuse the mask data into the audio feature data; The audio feature data to which the mask data is added is input into the front-end convolution module, and the audio feature data to which the mask data is integrated is converted into the marked audio feature data using the front-end convolution module.

6. The method according to claim 4, wherein: The tagging structure includes a fusion module and a front-end convolution module, and the audio feature data and the mask data are associated and input into the tagging structure to obtain the tagged audio feature data, including: Inputting the audio feature data into the front-end convolution module, and using the front-end convolution module to convert the audio feature data into a feature extraction result having time and frequency characteristics; The feature extraction result and the mask data are input into the fusion module, and the mask data is fused into the feature extraction result by using the fusion module to obtain the marked audio feature data.

7. A voiceprint coding network training method, wherein the voiceprint coding network comprises an audio feature extraction structure, a labeling structure and a main network structure; the method comprises: Acquire audio training data and mask data corresponding to the audio training data; wherein the mask data is used to characterize the application intention of the audio training data; Using the audio feature extraction structure, extracting features from the audio training data to obtain audio feature data; Associating the audio feature data and the mask data and inputting them into the marking structure to obtain marked audio feature data, wherein the marked audio feature data is a feature having time and frequency characteristics; Inputting the marked audio feature data into the main network to obtain a voiceprint code; Based on the voiceprint coding and a predetermined loss function, the voiceprint coding network is adjusted to obtain a trained voiceprint coding network.

8. An electronic device comprising a memory, a processor and a computer program stored in the memory, wherein the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

9. A computer-readable storage medium, wherein a computer program is stored in the computer-readable storage medium, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

10. A computer program product, wherein: The computer program product comprises a computer program, and when the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images