Speaker separation method and device, electronic equipment and storage medium

Abstract

The embodiment of the application provides a speaker separation method and device, electronic equipment and a storage medium, and belongs to the field of artificial intelligence and financial technology. The method comprises the following steps: obtaining a speaker separation sample containing at least two sample speaking objects; training an original voiceprint feature extraction model, an original accent feature extraction model and an original speaker separation model based on the speaker separation sample, obtaining a target voiceprint feature extraction model, a target accent feature extraction model and a target speaker separation model, performing speaker separation annotation on target voice data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model, obtaining target speaker annotation information, and performing voice separation on the target voice data based on the target speaker annotation information. The embodiment of the application can accurately extract the voiceprints and voice features of different speakers in voice data, thereby realizing accurate speaker separation and improving the accuracy of speaker separation.

Description

Technical Field

[0001] This application relates to the fields of artificial intelligence and financial technology, and in particular to a speaker separation method and apparatus, electronic device and storage medium. Background Technology

[0002] Speaker separation is an artificial intelligence technology that can independently separate the voice of each speaker from voice data containing multiple speakers. It can be applied to various scenarios, such as in the financial field, where speaker separation can be performed on voice data between customers and customer service representatives to filter out the customer's voice data for processing such as sentiment analysis and intent recognition.

[0003] Currently, speaker separation methods mainly extract and analyze the speaker's voiceprint features. However, when multiple speakers have similar voiceprint features, the accuracy of speaker separation cannot be guaranteed.

[0004] Therefore, improving the accuracy of speaker separation has become an urgent technical problem to be solved. Summary of the Invention

[0005] The main objective of this application is to provide a speaker separation method, apparatus, electronic device, and storage medium, which aims to improve the accuracy of speaker separation.

[0006] To achieve the above objectives, a first aspect of this application provides a speaker separation method, the method comprising:

[0007] Obtain speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0008] The sample speech data is subjected to spectral conversion to obtain sample spectral data;

[0009] Based on the sample spectrum data and the sample speaker annotation information, the preset original voiceprint feature extraction model is trained to obtain the target voiceprint feature extraction model.

[0010] Based on the sample spectrum data and the sample speaker annotation information, the preset original accent feature extraction model is trained to obtain the target accent feature extraction model;

[0011] The preset original speaker separation model is trained based on the sample spectrum data and the sample speaker annotation information to obtain the target speaker separation model;

[0012] The target speech data is separated and labeled by the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker labeling information; wherein, the target speech data contains at least two target speaking objects, and the target speaker labeling information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data;

[0013] Based on the target speaker annotation information, the target speech data is separated to obtain the target speaker speech data for each target speaking object.

[0014] In some embodiments, the step of training a preset original voiceprint feature extraction model based on the sample spectral data and the sample speaker annotation information to obtain a target voiceprint feature extraction model includes:

[0015] The original voiceprint features are obtained by extracting voiceprint features from the sample spectral data using the original voiceprint feature extraction model.

[0016] The original voiceprint features are downsampled to obtain sample voiceprint features;

[0017] Speaker prediction is performed based on the sample voiceprint features to obtain the first predicted speaker annotation information;

[0018] Loss calculation is performed based on the sample speaker annotation information and the first predicted speaker annotation information to obtain voiceprint prediction loss data;

[0019] The original voiceprint feature extraction model is optimized based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

[0020] In some embodiments, the step of training a preset original accent feature extraction model based on the sample spectral data and the sample speaker annotation information to obtain a target accent feature extraction model includes:

[0021] The original accent features are obtained by extracting accent features from the sample spectral data using the original accent feature extraction model.

[0022] The original accent features are downsampled to obtain sample accent features;

[0023] Speaker prediction is performed based on the accent features of the sample to obtain second predicted speaker annotation information;

[0024] Based on the sample speaker annotation information and the second predicted speaker annotation information, loss calculation is performed to obtain accent prediction loss data;

[0025] The original accent feature extraction model is optimized based on the accent prediction loss data to obtain the target accent feature extraction model.

[0026] In some embodiments, the step of training a preset original speaker separation model based on the sample spectral data and the sample speaker annotation information to obtain a target speaker separation model includes:

[0027] The original speaker separation features are obtained by extracting speaker features from the sample spectral data using the original speaker separation model.

[0028] The original speaker separation features are downsampled to obtain sample speaker separation features;

[0029] The sample voiceprint features, the sample accent features, and the sample speaker separation features are concatenated to obtain sample speaker separation data;

[0030] Attention features are obtained by performing attention calculations on the speaker separation data of the samples.

[0031] Speaker separation is performed on the attention features of the samples to obtain third predicted speaker annotation information;

[0032] Loss is calculated based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data;

[0033] The original speaker separation model is optimized based on the separation prediction loss data to obtain the target speaker separation model.

[0034] In some embodiments, the step of performing speaker separation and annotation on the pre-acquired target speech data using the target voiceprint feature extraction model, the target accent feature extraction model, and the target speaker separation model to obtain target speaker annotation information includes:

[0035] The target speech data is subjected to spectrum conversion to obtain target speech spectrum data;

[0036] The target voiceprint features are obtained by extracting voiceprint features from the target speech spectrum data using the target voiceprint feature extraction model.

[0037] The target accent features are obtained by extracting accent features from the target speech spectrum data using the target accent feature extraction model.

[0038] The target speaker separation features are obtained by extracting speaker features from the target speech spectrum data using the target speaker separation model.

[0039] The target voiceprint features, the target accent features, and the target speaker separation features are concatenated to obtain target speaker separation data;

[0040] Speaker separation is performed based on the target speaker separation data to obtain target speaker annotation information.

[0041] In some embodiments, the step of performing speaker separation based on the target speaker separation data to obtain target speaker annotation information includes:

[0042] Attention features are obtained by performing attention calculations on the target speaker separation data.

[0043] Speaker separation is performed on the target attention features to obtain the target speaker annotation information.

[0044] In some embodiments, the step of performing speech separation on the target speech data based on the target speaker annotation information to obtain target speaker speech data for each target speaking object includes:

[0045] For each target speaker, the speaker's speech time segment is obtained based on the target speaker annotation information;

[0046] Based on the speaker's speech period, speech extraction is performed on the target speech data to obtain the target speaker's speech data for each target speaking object.

[0047] To achieve the above objectives, a second aspect of this application provides a speaker separation device, the device comprising:

[0048] The sample data acquisition module is used to acquire speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0049] The sample spectrum conversion module is used to perform spectrum conversion on the sample speech data to obtain sample spectrum data;

[0050] The voiceprint feature extraction model training module is used to train the preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain the target voiceprint feature extraction model.

[0051] The accent feature extraction model training module is used to train the preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain the target accent feature extraction model.

[0052] The speaker separation model training module is used to train a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model.

[0053] The target speaker annotation module is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein, the target speech data contains at least two target speaking objects, and the target speaker annotation information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data;

[0054] The speaker speech separation module is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data for each target speaking object.

[0055] To achieve the above objectives, a third aspect of this application provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the method described in the first aspect.

[0056] To achieve the above objectives, a fourth aspect of the present application provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the method described in the first aspect.

[0057] The speaker separation method, apparatus, electronic device, and storage medium proposed in this application train the original voiceprint feature extraction model, the original accent feature extraction model, and the original speaker separation model using sample data from multiple speaking subjects. This improves the learning ability of the models, enabling them to accurately extract the voiceprints and speech features of different speakers in the target speech data, thereby achieving accurate speaker separation and improving the accuracy of speaker separation. Attached Figure Description

[0058] Figure 1 This is a flowchart of the speaker separation method provided in the embodiments of this application;

[0059] Figure 2 yes Figure 1 The flowchart of step S103 in the process;

[0060] Figure 3 yes Figure 1 The flowchart of step S104 in the process;

[0061] Figure 4 yes Figure 1 The flowchart of step S105 in the process;

[0062] Figure 5 yes Figure 1 The flowchart of step S106 in the process;

[0063] Figure 6 yes Figure 5 The flowchart of step S506 in the process;

[0064] Figure 7 yes Figure 1 The flowchart of step S107 in the process;

[0065] Figure 8 This is a schematic diagram of the speaker separation device provided in the embodiments of this application;

[0066] Figure 9 This is a schematic diagram of the hardware structure of the electronic device provided in the embodiments of this application. Detailed Implementation

[0067] To make the objectives, technical solutions, and advantages of this application clearer, the following detailed description is provided in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are merely illustrative and not intended to limit the scope of this application.

[0068] It should be noted that although functional modules are divided in the device schematic diagram and a logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first," "second," etc., in the specification, claims, and the aforementioned drawings are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this application belongs. The terminology used herein is for the purpose of describing embodiments of this application only and is not intended to limit this application.

[0070] First, let's analyze some of the terms used in this application:

[0071] Artificial Intelligence (AI): Artificial intelligence is a comprehensive technical science that aims to develop, research, apply, and maintain theories, methods, technologies, and application systems that can simulate, extend, and expand human intelligence. Simply put, AI is the science that enables machines to perform complex tasks that typically require human intelligence. These tasks include, but are not limited to, learning (acquiring new knowledge from data), reasoning (drawing conclusions from known information), self-correction (adjusting strategies based on feedback), natural language processing (understanding and generating human language), computer vision (recognizing and understanding objects in images and videos), and expert systems (mimicking the decision-making process of human experts in specific domains). The research areas of AI are broad, including machine learning, deep learning, computer vision, natural language processing, robotics, and expert systems.

[0072] Speaker separation is an audio processing technique that aims to separate the speech of each speaker independently from a mixed audio signal containing multiple speakers speaking simultaneously. This makes the speech signals of each speaker independent in time and space, facilitating subsequent speech recognition, voiceprint recognition, or other audio analysis tasks.

[0073] Speaker separation technology can be applied to a variety of scenarios. For example, in the financial field, it can separate the voice data between customers and customer service representatives, and filter out the customer's voice data for processing such as sentiment analysis and intent recognition.

[0074] Currently, speaker separation methods mainly extract and analyze the speaker's voiceprint features. However, when multiple speakers have similar voiceprint features, the accuracy of speaker separation cannot be guaranteed.

[0075] Based on this, embodiments of this application provide a speaker separation method and apparatus, an electronic device and a storage medium, aimed at improving the accuracy of speaker separation.

[0076] The speaker separation method, apparatus, electronic device, and storage medium provided in this application are specifically described through the following embodiments. First, the speaker separation method in this application is described.

[0077] The embodiments of this application can acquire and process relevant data based on artificial intelligence technology. Artificial intelligence (AI) is the theory, method, technology, and application system that uses digital computers or machines controlled by digital computers to simulate, extend, and expand human intelligence, perceive the environment, acquire knowledge, and use that knowledge to obtain optimal results.

[0078] Foundational technologies for artificial intelligence generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing, operating / interactive systems, and mechatronics. AI software technologies mainly encompass computer vision, robotics, biometrics, speech processing, natural language processing, and machine learning / deep learning.

[0079] The speaker separation method provided in this application relates to the fields of artificial intelligence and financial technology. The speaker separation method provided in this application can be applied to a terminal, a server, or software running on either a terminal or a server. In some embodiments, the terminal can be a smartphone, tablet, laptop, desktop computer, etc.; the server can be configured as an independent physical server, a server cluster or distributed system composed of multiple physical servers, or a cloud server providing basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application implementing the speaker separation method, but is not limited to the above forms.

[0080] This application can be used in a wide variety of general-purpose or special-purpose computer system environments or configurations. Examples include: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, and distributed computing environments including any of the above systems or devices. This application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc., that perform specific tasks or implement specific abstract data types. This application can also be practiced in distributed computing environments where tasks are performed by remote processing devices connected via a communication network. In distributed computing environments, program modules can reside in local and remote computer storage media, including storage devices.

[0081] It should be noted that in all specific embodiments of this application, when processing data related to user identity or characteristics, such as user information, user behavior data, user historical data, and user location information, user permission or consent is obtained first. Furthermore, the collection, use, and processing of this data comply with relevant laws, regulations, and standards. In addition, when embodiments of this application require access to sensitive personal information of users, separate permission or consent from the user is obtained through pop-ups or redirection to confirmation pages. Only after obtaining the user's separate permission or consent is the necessary user-related data required for the proper functioning of these embodiments acquired.

[0082] Figure 1 This is an optional flowchart of the speaker separation method provided in the embodiments of this application. Figure 1 The method may include, but is not limited to, steps S101 to S107.

[0083] Step S101: Obtain speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0084] Step S102: Perform spectrum conversion on the sample speech data to obtain sample spectrum data;

[0085] Step S103: Based on the sample spectrum data and the sample speaker annotation information, the preset original voiceprint feature extraction model is trained to obtain the target voiceprint feature extraction model.

[0086] Step S104: Based on the sample spectrum data and the sample speaker annotation information, the preset original accent feature extraction model is trained to obtain the target accent feature extraction model.

[0087] Step S105: Based on the sample spectrum data and sample speaker annotation information, train the preset original speaker separation model to obtain the target speaker separation model;

[0088] Step S106: The target speech data is separated and labeled by the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain the target speaker labeling information; wherein, the target speech data contains at least two target speaking objects, and the target speaker labeling information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data;

[0089] Step S107: Based on the target speaker annotation information, perform speech separation on the target speech data to obtain the target speaker speech data for each target speaking object.

[0090] Steps S101 to S107 as shown in the embodiments of this application improve the learning ability of the original voiceprint feature extraction model, the original accent feature extraction model, and the original speaker separation model by using sample data of multiple speaking subjects. This enables the accurate extraction of voiceprints and speech features of different speakers in the speech data, thereby achieving accurate speaker separation and improving the accuracy of speaker separation.

[0091] It should be noted that the speaker separation sample is a multi-speaker speech sample. The speaker separation sample has at least two sample speaking objects. The speaker separation sample includes sample speech data and sample speaker annotation information. The sample speech data includes multiple sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which each sample speech segment belongs.

[0092] Prior to step S101 in some embodiments, the speaker separation method further includes, but is not limited to, the following steps:

[0093] Acquire training speech data;

[0094] Data augmentation processing is performed on the training speech data to obtain sample speech data;

[0095] Annotation is generated from the sample speech data to obtain the speaker annotation information.

[0096] Speaker-separated samples are obtained by combining sample speech data and sample speaker annotation data.

[0097] Specifically, the step "perform data augmentation on the training speech data to obtain sample speech data" includes acquiring speech segments from different speakers and speech segments with silence / noise, and randomly splicing multiple speech segments into the training speech data to obtain sample speech data.

[0098] The random splicing is based on the following rules:

[0099] Two adjacent speech segments cannot be two silent / noisy speech segments;

[0100] Two adjacent speech segments cannot be from the same speaker;

[0101] Two adjacent speech segments cannot overlap.

[0102] In step S102 of some embodiments, the sample speech data can be spectrum converted using the Fbank method; alternatively, it can be done using methods such as Fast Fourier Transform (FFT), Short-Time Fourier Transform (STFT), and Mel-frequency cepstral coefficients (MFCC). The specific method chosen depends on the actual application scenario and is not limited to these methods.

[0103] In some embodiments, the number of sample spectra is Mel spectrum data with a preset data dimension, and the preset data dimension needs to be determined according to the input parameter requirements of the original voiceprint feature extraction model, the original accent feature extraction model and the original speaker separation model, and is not limited thereto.

[0104] Please see Figure 2 In some embodiments, step S103 may include, but is not limited to, steps S201 to S205:

[0105] Step S201: Extract voiceprint features from the sample spectral data using the original voiceprint feature extraction model to obtain the original voiceprint features;

[0106] Step S202: Downsample the original voiceprint features to obtain sample voiceprint features;

[0107] Step S203: Based on the sample voiceprint features, predict the speaker to obtain the first predicted speaker annotation information;

[0108] Step S204: Based on the sample speaker annotation information and the first predicted speaker annotation information, calculate the loss to obtain the voiceprint prediction loss data;

[0109] Step S205: Optimize the original voiceprint feature extraction model based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

[0110] Steps S201 to S205 as shown in the embodiments of this application first extract the original voiceprint features of the sample spectrum data through the original voiceprint feature extraction model, and obtain the voiceprint information of each sample speaker; the original voiceprint features are downsampled, which not only retains the key voiceprint information but also reduces the amount of computation. The downsampled sample voiceprint features are used to predict the speaker, and loss is calculated with the sample speaker annotation information to obtain voiceprint prediction loss data, which is used to optimize the original voiceprint feature extraction model, improve the model's ability to recognize speaker voiceprints and its robustness, improve the accuracy of voiceprint feature extraction, and provide a solid foundation for subsequent speaker separation.

[0111] In some embodiments, the original voiceprint extraction model is a model capable of extracting the speaker's voiceprint features from spectral data. Specifically, it can be an ECAPA-TDNN model, a ResNetSE model, an ERes2Net model, etc., and is not limited to these.

[0112] In some embodiments, step S202 may include, but is not limited to, the following steps:

[0113] Based on the preset voiceprint pooling window length, the original voiceprint features are processed into frames to obtain multiple sample voiceprint frame features and sample voiceprint frame extraction order.

[0114] The average frame features of each sample's voiceprint are calculated to obtain the average frame features of multiple samples' voiceprints.

[0115] Based on the sample voiceprint frame extraction order, the average frame features of multiple sample voiceprints are concatenated to obtain the sample voiceprint features.

[0116] It should be noted that the speaker pooling window length is used to compress speaker features, reducing the size of the feature map and thus reducing the number of parameters in subsequent processing, thereby improving computational efficiency. Specifically, the dimension of the speaker pooling window length needs to be determined based on the data dimension of the original speaker features and the input and output of the model, and is not limited to this.

[0117] In some embodiments, downsampling can also be achieved by max pooling the original voiceprint features.

[0118] It should be noted that, since speech data is continuous, the unit of the original voiceprint features can be at the frame level.

[0119] In step S203 of some embodiments, the original voiceprint features of each frame are used to predict the speaker through a preset first speaker predictor, thereby outputting the first predicted speaker label for each frame. Then, the first predicted speaker labels of all frames are summarized to obtain the first predicted speaker label information. The first predicted speaker label information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0120] Specifically, the speech segment information includes the start and end times of the speech segment.

[0121] In one embodiment, the preset first speaker predictor is a pre-trained speaker recognition model, which can be constructed from network structures such as AAM-Softmax, KMeans-Subcenter, and Inter-TopK.

[0122] In step S204 of some embodiments, loss data for each frame is obtained by performing loss calculation on the first predicted speaker annotation and sample speaker annotation information for each frame; the loss data of each frame is summed to obtain voiceprint prediction loss data.

[0123] Among them, the spectral loss can be calculated using loss functions such as mean squared error loss function, cross-entropy loss function, likelihood loss function, adversarial loss function, etc., and is not limited to these.

[0124] In step S205 of some embodiments, methods such as backpropagation, gradient descent, and momentum update can be used to optimize the original voiceprint feature extraction model based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

[0125] Please see Figure 3 In some embodiments, step S104 may include, but is not limited to, steps S301 to S305:

[0126] Step S301: Extract accent features from the sample spectral data using the original accent feature extraction model to obtain the original accent features;

[0127] Step S302: Downsample the original accent features to obtain sample accent features;

[0128] Step S303: Based on the accent features of the sample, predict the speaker to obtain the second predicted speaker annotation information;

[0129] Step S304: Based on the sample speaker annotation information and the second predicted speaker annotation information, calculate the loss to obtain accent prediction loss data;

[0130] Step S305: Optimize the original accent feature extraction model based on the accent prediction loss data to obtain the target accent feature extraction model.

[0131] Steps S301 to S305 of this embodiment involve extracting the original accent features of the sample spectral data using the original accent feature extraction model to obtain the accent information of each speaker. The original accent features are then downsampled, which preserves key accent information while reducing computation. The downsampled accent features are used to predict the speaker, and the loss is calculated by comparing them with the labeled information of the sample speakers to obtain accent prediction loss data. This data is then used to optimize the original accent feature extraction model, improving the model's ability to recognize and its robustness in speaker accents, thereby enhancing the accuracy of accent feature extraction and providing a solid foundation for subsequent speaker separation.

[0132] In some embodiments, the original accent extraction model is a model that can extract the accent features of a speaker from spectral data. Specifically, it can be an ECAPA-TDNN model, or a CNN model, RNN model, LSTM model, etc., and is not limited thereto.

[0133] In some embodiments, step S302 may include, but is not limited to, the following steps:

[0134] Based on the preset accent pooling window length, the original accent features are processed into frames to obtain multiple sample accent frame features and sample accent frame extraction order.

[0135] The average frame features of each sample accent are calculated to obtain the average frame features of multiple sample accents.

[0136] Based on the order of sample accent frame extraction, the average frame features of multiple sample accents are concatenated to obtain the sample accent features.

[0137] It should be noted that the accent pooling window length is used to compress accent features, which can reduce the size of the feature map, thereby reducing the number of parameters in subsequent processing and improving computational efficiency. Specifically, the dimension of the accent pooling window length needs to be determined based on the data dimension of the original accent features and the input and output of the model, and is not limited to this.

[0138] In some embodiments, downsampling can also be achieved by max pooling the original accent features.

[0139] It should be noted that, since speech data is continuous, the unit for original accent features can be at the frame level.

[0140] In step S303 of some embodiments, the original accent features of each frame are used to predict the speaker through a preset second speaker predictor, thereby outputting the second predicted speaker annotation for each frame. Then, the second predicted speaker annotations of all frames are summarized to obtain the second predicted speaker annotation information. The second predicted speaker annotation information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0141] Specifically, the speech segment information includes the start and end times of the speech segment.

[0142] In one embodiment, the preset second speaker predictor is a pre-trained speaker recognition model, which may consist of fully connected layers and is used to classify the speaking object. The output category is the accent or dialect category, thus corresponding to different speaking objects.

[0143] In step S304 of some embodiments, loss data for each frame is obtained by performing loss calculation on the second predicted speaker annotation and sample speaker annotation information for each frame; the loss data of each frame is summed to obtain accent prediction loss data.

[0144] Among them, the spectral loss can be calculated using loss functions such as mean squared error loss function, cross-entropy loss function, likelihood loss function, adversarial loss function, etc., and is not limited to these.

[0145] In step S305 of some embodiments, methods such as backpropagation, gradient descent, and momentum update can be used to optimize the original accent feature extraction model based on accent prediction loss data to obtain the target accent feature extraction model.

[0146] Please see Figure 4 In some embodiments, step S105 may include, but is not limited to, steps S401 to S407:

[0147] Step S401: Extract speaker features from the sample spectral data using the original speaker separation model to obtain the original speaker separation features;

[0148] Step S402: Downsample the original speaker separation features to obtain sample speaker separation features;

[0149] Step S403: Perform feature concatenation on the sample voiceprint features, sample accent features, and sample speaker separation features to obtain sample speaker separation data;

[0150] Step S404: Perform attention calculation on the speaker separation data of the samples to obtain the sample attention features;

[0151] Step S405: Speaker separation is performed on the sample attention features to obtain the third predicted speaker annotation information;

[0152] Step S406: Calculate the loss based on the sample speaker annotation information and the third predicted speaker annotation information to obtain the separation prediction loss data;

[0153] Step S407: Optimize the original speaker separation model based on the separation prediction loss data to obtain the target speaker separation model.

[0154] Steps S401 to S407, as illustrated in this embodiment, involve extracting speaker features from the sample spectral data using the original speaker separation model. This yields the original speaker separation features for each sample speaker, including information such as the speaker's voiceprint, accent, timbre, and pitch. The original speaker separation features are downsampled, preserving key speaker information while reducing computation. The downsampled sample speaker separation data is then concatenated with the sample voiceprint and accent features to obtain sample speaker separation data. Attention is calculated on the sample speaker separation data to obtain sample attention features. Speaker separation is then performed on the sample attention features to obtain third predicted speaker annotation information. Loss calculation is performed based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data. This loss data is used to optimize the original speaker separation model, improving its ability to recognize speaker features and its robustness, thereby enhancing the accuracy of speaker feature extraction and providing a solid foundation for subsequent speaker separation.

[0155] In some embodiments, the original voiceprint extraction model is a model capable of extracting speaker separation features from spectral data. Specifically, it can be a Conformer model, or a Branchformer model, an EfficientConformer model, a Squeezeformer model, etc., and is not limited thereto.

[0156] In some embodiments, step S402 may include, but is not limited to, the following steps:

[0157] Based on the preset speaker separation pooling window length, the original speaker separation features are processed into frames to obtain multiple sample speaker frame features and sample speaker frame extraction order.

[0158] The average frame features of each speaker sample are calculated to obtain the average frame features of multiple speaker samples.

[0159] Based on the frame extraction order of the sample speakers, the average frame features of multiple sample speakers are concatenated to obtain the sample speaker separation features.

[0160] It should be noted that the speaker separation pooling window length is used to compress speaker separation features, which can reduce the size of the feature map, thereby reducing the number of parameters in subsequent processing and improving computational efficiency. Specifically, the dimension of the speaker separation pooling window length needs to be determined based on the data dimension of the original speaker separation features and the input and output of the model, and is not limited to this.

[0161] In some embodiments, downsampling can also be achieved by max pooling the original speaker separation features.

[0162] It should be noted that the lengths of the voiceprint pooling window, accent pooling window, and speaker separation pooling window can be set to the same or different, and are not limited to this.

[0163] In step S403 of some embodiments, since the sample voiceprint features, sample accent features, and sample speaker separation features are all in vector form, feature concatenation involves sequentially concatenating the sample voiceprint features, sample accent features, and sample speaker separation features into a single feature vector, i.e.:

[0164] Sample speaker separation data = [Sample voiceprint features, Sample accent features, Sample speaker separation features].

[0165] In step S404 of some embodiments, attention is calculated on the sample speaker separation data through a Transformer Encoder layer with a self-attention mechanism to obtain sample attention features.

[0166] Specifically, step S404 may include, but is not limited to, the following steps:

[0167] A matrix transformation is performed on the sample speaker separation data to obtain the sample speaker separation transformation matrix; wherein, the sample speaker separation transformation matrix includes the sample query transformation matrix, the sample key transformation matrix, and the sample value transformation matrix;

[0168] The sample attention score is obtained by aggregating and calculating the sample query transformation matrix and the sample key transformation matrix.

[0169] The sample attention scores are normalized to obtain the sample attention weights;

[0170] The sample attention features are obtained by aggregating and calculating the sample attention weights and sample value transformation matrices.

[0171] By introducing an attention mechanism to learn and capture the dependencies between different parts of speaker separation features, and assigning different weights to different parts of speaker separation features accordingly, the model's ability to learn speaker separation features is improved, which helps to improve the accuracy of speaker separation.

[0172] Specifically, the sample attention weights are obtained by calculating the dot product of the sample query transformation matrix and the sample key transformation matrix, and then applying the softmax function for normalization calculation along the column dimension (usually the sequence length dimension). These weights reflect the correlation strength of different parts of the speaker separation features.

[0173] The sample attention features are obtained by weighting the sample attention weights and the sample value transformation matrix.

[0174] In step S405 of some embodiments, the speaker is separated from the sample attention features by a preset speaker prediction model to obtain third predicted speaker annotation information; wherein, the third predicted speaker annotation information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0175] In one embodiment, a pre-defined speaker prediction model can classify speakers based on sample attention features, thereby predicting different speaking objects. Specifically, the speaker prediction model can consist of fully connected layers. The output category is the category probability of each speaking object, thus corresponding to different speaking objects.

[0176] In step S406 of some embodiments, loss calculation is performed on the sample speaker annotation information and the third predicted speaker annotation information to obtain separate prediction loss data;

[0177] In some embodiments, the loss calculation process includes calculating the permutation invariance loss function and the concordance correlation coefficient loss function;

[0178] By aggregating the permutation invariance loss data and the consistency correlation coefficient loss data, the separation prediction loss data is obtained.

[0179] In step S407 of some embodiments, methods such as backpropagation, gradient descent, and momentum update can be used to optimize the original speaker separation model based on the separation prediction loss data to obtain the target speaker separation model.

[0180] Please see Figure 5 In some embodiments, step S106 may also include, but is not limited to, steps S501 to S506:

[0181] Step S501: Perform spectrum conversion on the target speech data to obtain target speech spectrum data;

[0182] Step S502: Extract voiceprint features from the target speech spectrum data using the target voiceprint feature extraction model to obtain the target voiceprint features;

[0183] Step S503: Extract accent features from the target speech spectrum data using the target accent feature extraction model to obtain the target accent features;

[0184] Step S504: Extract speaker features from the target speech spectrum data using the target speaker separation model to obtain the target speaker separation features;

[0185] Step S505: Perform feature concatenation on the target voiceprint features, target accent features, and target speaker separation features to obtain target speaker separation data;

[0186] Step S506: Speaker separation is performed based on the target speaker separation data to obtain the target speaker annotation information.

[0187] Steps S501 to S506 of the embodiments of this application involve converting the target speech data into a spectrum to obtain target speech spectrum data. Then, the target speech spectrum data is processed using a target voiceprint feature extraction model, a target accent feature extraction model, and a target speaker separation model, respectively. This accurately extracts target voiceprint features, target accent features, and target speaker separation features from the target speech spectrum data. Furthermore, the target voiceprint features, target accent features, and target speaker separation features are concatenated to obtain target speaker separation data. This integrates multi-dimensional feature information, enhancing the data's ability to represent speaker identity. Finally, speaker separation is performed based on the target speaker separation data to obtain target speaker annotation information, thus improving the accuracy of speaker separation.

[0188] In step S501 of some embodiments, the target speech data can be spectrum converted using the Fbank method; alternatively, it can be done using methods such as Fast Fourier Transform (FFT), Short-Time Fourier Transform (STFT), and Mel-frequency cepstral coefficients (MFCC). The specific method chosen depends on the actual application scenario and is not limited to these methods.

[0189] In some embodiments, the number of sample spectra is Mel spectrum data with a preset data dimension, and the preset data dimension needs to be determined according to the input parameter requirements of the target voiceprint feature extraction model, the target accent feature extraction model, and the target speaker separation model, and is not limited thereto.

[0190] In some embodiments, step S502 includes, but is not limited to, the following steps:

[0191] The target speech spectrum data is used to extract voiceprint features through the target voiceprint feature extraction model to obtain the initial voiceprint features.

[0192] The initial voiceprint features are downsampled to obtain the target voiceprint features.

[0193] Specifically, the step "downsampling the initial voiceprint features to obtain the target voiceprint features" may include, but is not limited to, the following steps:

[0194] Based on the preset voiceprint pooling window length, the initial voiceprint features are processed into frames to obtain multiple initial voiceprint frame features and the initial voiceprint frame extraction order.

[0195] The average of each initial voiceprint frame feature is calculated to obtain multiple initial voiceprint average frame features.

[0196] Based on the initial voiceprint frame extraction order, the average frame features of multiple initial voiceprints are concatenated to obtain the target voiceprint features.

[0197] In some embodiments, downsampling can also be achieved by max pooling the initial voiceprint features.

[0198] In some embodiments, step S503 includes, but is not limited to, the following steps:

[0199] The initial accent features are obtained by extracting accent features from the target speech spectrum data using a target accent feature extraction model.

[0200] The initial accent features are downsampled to obtain the target accent features.

[0201] Specifically, the step "downsampling the initial accent features to obtain the target accent features" may include, but is not limited to, the following steps:

[0202] The initial accent features are segmented into frames based on the preset accent pooling window length, resulting in multiple initial accent frame features and the initial accent frame extraction order.

[0203] The average of each initial accent frame feature is calculated to obtain multiple initial accent average frame features;

[0204] Based on the initial accent frame extraction order, the average frame features of multiple initial accents are concatenated to obtain the target accent features.

[0205] In some embodiments, downsampling can also be achieved by max pooling the initial accent features.

[0206] In some embodiments, step S504 may include, but is not limited to, the following steps:

[0207] Initial speaker separation features are obtained by extracting speaker features from the target speech spectrum data using a target speaker separation model.

[0208] The initial speaker separation features are downsampled to obtain the target speaker separation features.

[0209] Specifically, the step "downsampling the initial speaker separation features to obtain the target speaker separation features" may include, but is not limited to, the following steps:

[0210] Based on the preset speaker separation pooling window length, the initial speaker separation features are processed into frames to obtain multiple initial speaker frame features and the initial speaker frame extraction order.

[0211] The average frame features of each initial speaker are calculated to obtain multiple average frame features of the initial speaker.

[0212] Based on the initial speaker frame extraction order, the average frame features of multiple initial speakers are concatenated to obtain the target speaker separation features.

[0213] In some embodiments, downsampling can also be achieved by max pooling the initial speaker separation features.

[0214] In step S505 of some embodiments, since the target voiceprint features, target accent features, and target speaker separation features are all in vector form, feature concatenation involves sequentially concatenating the target voiceprint features, target accent features, and target speaker separation features into a single feature vector, i.e.:

[0215] Target speaker separation data = [target voiceprint features, target accent features, target speaker separation features].

[0216] Please see Figure 6 In some embodiments, step S506 includes, but is not limited to, steps S601 to S602:

[0217] Step S601: Perform attention calculation on the target speaker separation data to obtain target attention features;

[0218] Step S602: Speaker separation is performed on the target attention features to obtain the target speaker annotation information.

[0219] Steps S601 to S602 shown in the embodiments of this application, by performing attention calculation on the target speaker separation data, can focus on key feature regions during the speaker separation process. Therefore, speaker separation based on target attention features can make the speaker separation results more consistent with the actual context and speaker characteristics, thereby improving the accuracy of speaker separation.

[0220] In some embodiments, step S601 may include, but is not limited to, the following steps:

[0221] A matrix transformation is performed on the target speaker separation data to obtain the target speaker separation transformation matrix; wherein, the target speaker separation transformation matrix includes the target query transformation matrix, the target key transformation matrix, and the target value transformation matrix;

[0222] The target attention score is obtained by aggregating and calculating the target query transformation matrix and the target key transformation matrix.

[0223] The target attention score is normalized to obtain the target attention weight;

[0224] The target attention features are obtained by aggregating and calculating the target attention weights and the target value transformation matrix.

[0225] It should be noted that the specific implementation of step S601 is the same as that shown in step S404, and will not be repeated here.

[0226] Please see Figure 7 In some embodiments, step S107 may include, but is not limited to, steps S701 to S702:

[0227] Step S701: For each target speaker, obtain the speaker's speech time period based on the target speaker annotation information;

[0228] Step S702: Extract speech from the target speech data based on the speaker's speech time period to obtain the target speaker speech data for each target speaking object.

[0229] Steps S701 to S702, as shown in the embodiments of this application, parse the target speaker annotation information to obtain the speech segment of each target speaker in the target speech data, ensuring that the speech segment corresponding to each target speaker can be accurately extracted from the entire target speech data, thereby improving the accuracy of speaker separation.

[0230] It should be noted that since the target speech data contains at least two target speaking objects, the target speaker annotation information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data; correspondingly, the speech segment information is the speaker's speech time period for each target speaking object, including the start time and end time of the speech segment.

[0231] In step S702 of some embodiments, the target speaker voice data of each target speaker object is extracted from the target voice data according to the speaker's voice time period (the start and end time of the voice segment), thereby completing speaker separation.

[0232] Please see Figure 8 This application also provides a speaker separation device that can implement the above-described speaker separation method. The device includes:

[0233] The sample data acquisition module 801 is used to acquire speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0234] The sample spectrum conversion module 802 is used to perform spectrum conversion on the sample speech data to obtain sample spectrum data;

[0235] The voiceprint feature extraction model training module 803 is used to train the preset original voiceprint feature extraction model based on sample spectrum data and sample speaker annotation information to obtain the target voiceprint feature extraction model.

[0236] The accent feature extraction model training module 804 is used to train the preset original accent feature extraction model based on sample spectrum data and sample speaker annotation information to obtain the target accent feature extraction model.

[0237] The speaker separation model training module 805 is used to train the preset original speaker separation model based on sample spectrum data and sample speaker annotation information to obtain the target speaker separation model.

[0238] The target speaker annotation module 806 is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein, the target speech data contains at least two target speaking objects, and the target speaker annotation information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data;

[0239] The speaker speech separation module 807 is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data for each target speaking object.

[0240] The specific implementation of the speaker separation device is basically the same as the specific embodiment of the speaker separation method described above, and will not be repeated here.

[0241] This application also provides an electronic device, which includes a memory and a processor. The memory stores a computer program, and the processor executes the computer program to implement the speaker separation method described above. This electronic device can be any smart terminal, including tablet computers, in-vehicle computers, etc.

[0242] Please see Figure 9 , Figure 9 The hardware structure of an electronic device according to another embodiment is illustrated. The electronic device includes:

[0243] The processor 901 can be implemented using a general-purpose CPU (Central Processing Unit), microprocessor, application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of this application.

[0244] The memory 902 can be implemented as a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store the operating system and other application programs. When the technical solutions provided in the embodiments of this specification are implemented through software or firmware, the relevant program code is stored in the memory 902 and is called and executed by the processor 901 using the speaker separation method of the embodiments of this application.

[0245] The input / output interface 903 is used to implement information input and output;

[0246] The communication interface 904 is used to enable communication and interaction between this device and other devices. Communication can be achieved through wired means (such as USB, Ethernet cable, etc.) or wireless means (such as mobile network, WIFI, Bluetooth, etc.).

[0247] Bus 905 transmits information between various components of the device (e.g., processor 901, memory 902, input / output interface 903, and communication interface 904);

[0248] The processor 901, memory 902, input / output interface 903, and communication interface 904 are connected to each other within the device via bus 905.

[0249] This application also provides a computer-readable storage medium storing a computer program that, when executed by a processor, implements the above-described speaker separation method.

[0250] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs and non-transitory computer-executable programs. Furthermore, memory may include high-speed random access memory, and may also include non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory may optionally include memory remotely located relative to the processor, and these remote memories can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, intranets, local area networks, mobile communication networks, and combinations thereof.

[0251] The speaker separation method, apparatus, electronic device, and storage medium provided in this application embodiment train the original voiceprint feature extraction model, the original accent feature extraction model, and the original speaker separation model using sample data of multiple sample speaking objects. This improves the learning ability of the models, thereby enabling accurate extraction of voiceprints and speech features of different speakers in the target speech data, thus achieving accurate speaker separation and improving the accuracy of speaker separation.

[0252] The embodiments described in this application are for the purpose of more clearly illustrating the technical solutions of the embodiments of this application, and do not constitute a limitation on the technical solutions provided by the embodiments of this application. As those skilled in the art will know, with the evolution of technology and the emergence of new application scenarios, the technical solutions provided by the embodiments of this application are also applicable to similar technical problems.

[0253] Those skilled in the art will understand that the technical solutions shown in the figures do not constitute a limitation on the embodiments of this application, and may include more or fewer steps than shown, or combine certain steps, or different steps.

[0254] The device embodiments described above are merely illustrative. The units described as separate components may or may not be physically separate; that is, they may be located in one place or distributed across multiple network units. Some or all of the modules can be selected to achieve the purpose of this embodiment according to actual needs.

[0255] Those skilled in the art will understand that all or some of the steps in the methods disclosed above, as well as the functional modules / units in the systems and devices, can be implemented as software, firmware, hardware, or suitable combinations thereof.

[0256] The terms “first,” “second,” “third,” “fourth,” etc. (if present) in the specification and accompanying drawings of this application are used to distinguish similar objects and are not necessarily used to describe a specific order or sequence. It should be understood that such data can be interchanged where appropriate so that the embodiments of this application described herein can be implemented in orders other than those illustrated or described herein. Furthermore, the terms “comprising” and “having,” and any variations thereof, are intended to cover non-exclusive inclusion; for example, a process, method, system, product, or apparatus that comprises a series of steps or units is not necessarily limited to those steps or units explicitly listed, but may include other steps or units not explicitly listed or inherent to such processes, methods, products, or apparatus.

[0257] It should be understood that in this application, "at least one (item)" means one or more, and "more than" means two or more. "And / or" is used to describe the relationship between related objects, indicating that three relationships can exist. For example, "A and / or B" can represent three cases: only A exists, only B exists, and both A and B exist simultaneously, where A and B can be singular or plural. The character " / " generally indicates that the preceding and following related objects are in an "or" relationship. "At least one (item) of the following" or similar expressions refer to any combination of these items, including any combination of single or plural items. For example, at least one (item) of a, b, or c can represent: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, and c can be single or multiple.

[0258] In the several embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative; for instance, the division of the units described above is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0259] The units described above as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0260] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. The integrated unit can be implemented in hardware or as a software functional unit.

[0261] If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of this application, in essence, or the part that contributes to the prior art, or all or part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes multiple instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of the various embodiments of this application. The aforementioned storage medium includes various media capable of storing programs, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0262] The preferred embodiments of the present application have been described above with reference to the accompanying drawings, but this does not limit the scope of the claims of the present application. Any modifications, equivalent substitutions, and improvements made by those skilled in the art without departing from the scope and substance of the embodiments of the present application shall be within the scope of the claims of the present application.

Claims

1. A speaker separation method, characterized in that, The method includes: Obtain speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs; The sample speech data is subjected to spectral conversion to obtain sample spectral data; Based on the sample spectrum data and the sample speaker annotation information, the preset original voiceprint feature extraction model is trained to obtain the target voiceprint feature extraction model. Based on the sample spectrum data and the sample speaker annotation information, the preset original accent feature extraction model is trained to obtain the target accent feature extraction model; The preset original speaker separation model is trained based on the sample spectrum data and the sample speaker annotation information to obtain the target speaker separation model; The target speech data is separated and labeled by the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker labeling information; wherein, the target speech data contains at least two target speaking objects, and the target speaker labeling information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data; Based on the target speaker annotation information, the target speech data is separated to obtain the target speaker speech data for each target speaking object; The step of performing speaker separation and annotation on the pre-acquired target speech data using the target voiceprint feature extraction model, the target accent feature extraction model, and the target speaker separation model to obtain target speaker annotation information includes: The target speech data is subjected to spectrum conversion to obtain target speech spectrum data; The target voiceprint features are obtained by extracting voiceprint features from the target speech spectrum data using the target voiceprint feature extraction model. The target accent features are obtained by extracting accent features from the target speech spectrum data using the target accent feature extraction model. The target speaker separation features are obtained by extracting speaker features from the target speech spectrum data using the target speaker separation model. The target voiceprint features, the target accent features, and the target speaker separation features are concatenated to obtain target speaker separation data; Speaker separation is performed based on the target speaker separation data to obtain target speaker annotation information; The step of performing speaker separation based on the target speaker separation data to obtain target speaker annotation information includes: Self-attention calculation is performed on the target speaker separation data to obtain target attention features; Speaker separation is performed on the target attention features to obtain the target speaker annotation information.

2. The method according to claim 1, characterized in that, The step of training a preset original voiceprint feature extraction model based on the sample spectral data and the sample speaker annotation information to obtain a target voiceprint feature extraction model includes: The original voiceprint features are obtained by extracting voiceprint features from the sample spectral data using the original voiceprint feature extraction model. The original voiceprint features are downsampled to obtain sample voiceprint features; Speaker prediction is performed based on the sample voiceprint features to obtain the first predicted speaker annotation information; Loss calculation is performed based on the sample speaker annotation information and the first predicted speaker annotation information to obtain voiceprint prediction loss data; The original voiceprint feature extraction model is optimized based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

3. The method according to claim 2, characterized in that, The step of training a preset original accent feature extraction model based on the sample spectral data and the sample speaker annotation information to obtain a target accent feature extraction model includes: The original accent features are obtained by extracting accent features from the sample spectral data using the original accent feature extraction model. The original accent features are downsampled to obtain sample accent features; Speaker prediction is performed based on the accent features of the sample to obtain second predicted speaker annotation information; Based on the sample speaker annotation information and the second predicted speaker annotation information, loss calculation is performed to obtain accent prediction loss data; The original accent feature extraction model is optimized based on the accent prediction loss data to obtain the target accent feature extraction model.

4. The method according to claim 3, characterized in that, The step of training a preset original speaker separation model based on the sample spectral data and the sample speaker annotation information to obtain a target speaker separation model includes: The original speaker separation features are obtained by extracting speaker features from the sample spectral data using the original speaker separation model. The original speaker separation features are downsampled to obtain sample speaker separation features; The sample voiceprint features, the sample accent features, and the sample speaker separation features are concatenated to obtain sample speaker separation data; Attention features are obtained by performing attention calculations on the speaker separation data of the samples. Speaker separation is performed on the attention features of the samples to obtain third predicted speaker annotation information; Loss is calculated based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data; The original speaker separation model is optimized based on the separation prediction loss data to obtain the target speaker separation model.

5. The method according to any one of claims 1-4, characterized in that, The step of performing speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data for each target speaking object includes: For each target speaker, the speaker's speech time segment is obtained based on the target speaker annotation information; Based on the speaker's speech period, speech extraction is performed on the target speech data to obtain the target speaker's speech data for each target speaking object.

6. A speaker separation device, characterized in that, The device includes: The sample data acquisition module is used to acquire speaker separation samples containing at least two sample speaking objects; wherein, the speaker separation samples include sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs; The sample spectrum conversion module is used to perform spectrum conversion on the sample speech data to obtain sample spectrum data; The voiceprint feature extraction model training module is used to train the preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain the target voiceprint feature extraction model. The accent feature extraction model training module is used to train the preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain the target accent feature extraction model. The speaker separation model training module is used to train a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model. The target speaker annotation module is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein, the target speech data contains at least two target speaking objects, and the target speaker annotation information is used to characterize the speech segment information to which each target speaking object belongs in the target speech data; The speaker speech separation module is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data for each target speaking object. The step of performing speaker separation and annotation on the pre-acquired target speech data using the target voiceprint feature extraction model, the target accent feature extraction model, and the target speaker separation model to obtain target speaker annotation information includes: The target speech data is subjected to spectrum conversion to obtain target speech spectrum data; The target voiceprint features are obtained by extracting voiceprint features from the target speech spectrum data using the target voiceprint feature extraction model. The target accent features are obtained by extracting accent features from the target speech spectrum data using the target accent feature extraction model. The target speaker separation features are obtained by extracting speaker features from the target speech spectrum data using the target speaker separation model. The target voiceprint features, the target accent features, and the target speaker separation features are concatenated to obtain target speaker separation data; Speaker separation is performed based on the target speaker separation data to obtain target speaker annotation information; The step of performing speaker separation based on the target speaker separation data to obtain target speaker annotation information includes: Self-attention calculation is performed on the target speaker separation data to obtain target attention features; Speaker separation is performed on the target attention features to obtain the target speaker annotation information.

7. An electronic device, characterized in that, The electronic device includes a memory and a processor, the memory storing a computer program, and the processor executing the computer program to implement the method according to any one of claims 1 to 5.

8. A computer-readable storage medium storing a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 5.

Images