Speaker separation method and device, electronic equipment and storage medium

Abstract

The embodiment of the invention provides a speaker separation method and device, electronic equipment and a storage medium, and belongs to the technical field of artificial intelligence and finance. The method comprises the following steps: acquiring a speaker separation sample containing at least two sample speaking objects; training the original voiceprint feature extraction model, the original accent feature extraction model and the original speaker separation model based on the speaker separation sample to obtain a target voiceprint feature extraction model, a target accent feature extraction model and a target speaker separation model; and performing speaker separation labeling on the target voice data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker labeling information, and performing voice separation on the target voice data based on the target speaker labeling information. According to the embodiment of the invention, voiceprints and voice features of different speakers in the voice data can be accurately extracted, so that accurate speaker separation is realized, and the accuracy of speaker separation is improved.

Description

Technical Field

[0001] The present application relates to the fields of artificial intelligence and financial technology, and in particular to a speaker separation method and device, an electronic device and a storage medium. Background Art

[0002] Speaker separation is an artificial intelligence technology that can independently separate the speech of each speaker from speech data containing multiple speakers. It can be applied to a variety of scenarios. For example, in the financial field, speaker separation is performed on the speech data between customers and customer service, and the customer's speech data is screened out for processing such as sentiment analysis and intent recognition.

[0003] Currently, speaker separation methods mainly extract and analyze the voiceprint features of speakers. However, when multiple speakers have similar voiceprint features, the accuracy of speaker separation cannot be guaranteed.

[0004] Therefore, how to improve the accuracy of speaker separation has become a technical problem that needs to be solved urgently. Summary of the invention

[0005] The main purpose of the embodiments of the present application is to provide a speaker separation method and device, an electronic device and a storage medium, aiming to improve the accuracy of speaker separation.

[0006] To achieve the above object, a first aspect of an embodiment of the present application provides a method for speaker separation, the method comprising:

[0007] Acquire a speaker separation sample including at least two sample speaking objects; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0008] Performing spectrum conversion on the sample speech data to obtain sample spectrum data;

[0009] Based on the sample spectrum data and the sample speaker annotation information, a preset original voiceprint feature extraction model is trained to obtain a target voiceprint feature extraction model;

[0010] Performing model training on a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model;

[0011] Based on the sample spectrum data and the sample speaker annotation information, a preset original speaker separation model is trained to obtain a target speaker separation model;

[0012] The target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model are used to perform speaker separation and labeling on the pre-acquired target speech data to obtain target speaker labeling information; wherein the target speech data includes at least two target speakers, and the target speaker labeling information is used to characterize the speech segment information to which each target speaker belongs in the target speech data;

[0013] The target speech data is subjected to speech separation based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

[0014] In some embodiments, the performing model training on a preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target voiceprint feature extraction model includes:

[0015] Extracting voiceprint features from the sample spectrum data using the original voiceprint feature extraction model to obtain original voiceprint features;

[0016] Downsampling the original voiceprint features to obtain sample voiceprint features;

[0017] Perform speaker prediction based on the sample voiceprint feature to obtain first predicted speaker labeling information;

[0018] Performing loss calculation based on the sample speaker annotation information and the first predicted speaker annotation information to obtain voiceprint prediction loss data;

[0019] The original voiceprint feature extraction model is optimized based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

[0020] In some embodiments, the training of a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model includes:

[0021] Extracting accent features from the sample spectrum data using the original accent feature extraction model to obtain original accent features;

[0022] Downsampling the original accent feature to obtain a sample accent feature;

[0023] Perform speaker prediction based on the sample accent feature to obtain second predicted speaker annotation information;

[0024] Performing loss calculation based on the sample speaker annotation information and the second predicted speaker annotation information to obtain accent prediction loss data;

[0025] The original accent feature extraction model is optimized based on the accent prediction loss data to obtain the target accent feature extraction model.

[0026] In some embodiments, the performing model training on a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model includes:

[0027] Extracting speaker features from the sample spectrum data using the original speaker separation model to obtain original speaker separation features;

[0028] Downsampling the original speaker separation feature to obtain a sample speaker separation feature;

[0029] Performing feature splicing on the sample voiceprint feature, the sample accent feature and the sample speaker separation feature to obtain sample speaker separation data;

[0030] Performing attention calculation on the sample speaker separation data to obtain a sample attention feature;

[0031] Performing speaker separation on the sample attention features to obtain third predicted speaker labeling information;

[0032] Perform loss calculation based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data;

[0033] The original speaker separation model is optimized based on the separation prediction loss data to obtain the target speaker separation model.

[0034] In some embodiments, the speaker separation and annotation of the pre-acquired target speech data by using the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain the target speaker annotation information includes:

[0035] Performing spectrum conversion on the target speech data to obtain target speech spectrum data;

[0036] Performing voiceprint feature extraction on the target speech spectrum data using the target voiceprint feature extraction model to obtain target voiceprint features;

[0037] Extracting the target accent feature from the target speech spectrum data using the target accent feature extraction model to obtain a target accent feature;

[0038] Extracting speaker features from the target speech spectrum data using the target speaker separation model to obtain target speaker separation features;

[0039] Performing feature splicing on the target voiceprint feature, the target accent feature and the target speaker separation feature to obtain target speaker separation data;

[0040] Speaker separation is performed based on the target speaker separation data to obtain target speaker labeling information.

[0041] In some embodiments, performing speaker separation based on the target speaker separation data to obtain target speaker labeling information includes:

[0042] Performing attention calculation on the target speaker separation data to obtain a target attention feature;

[0043] Speaker separation is performed on the target attention feature to obtain the target speaker labeling information.

[0044] In some embodiments, performing speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker includes:

[0045] For each target speaker, obtaining the speaker's speech period according to the target speaker's annotation information;

[0046] The target voice data is subjected to voice extraction based on the speaker voice period to obtain the target speaker voice data of each target speaker.

[0047] To achieve the above-mentioned purpose, a second aspect of an embodiment of the present application provides a speaker separation device, the device comprising:

[0048] A sample data acquisition module, used to acquire a speaker separation sample containing at least two sample speaking objects; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs;

[0049] A sample spectrum conversion module, used for performing spectrum conversion on the sample speech data to obtain sample spectrum data;

[0050] A voiceprint feature extraction model training module is used to perform model training on a preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target voiceprint feature extraction model;

[0051] An accent feature extraction model training module, used for training a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model;

[0052] A speaker separation model training module, used for training a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model;

[0053] A target speaker annotation module is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein the target speech data contains at least two target speakers, and the target speaker annotation information is used to characterize the speech segment information to which each target speaker belongs in the target speech data;

[0054] The speaker speech separation module is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

[0055] To achieve the above objectives, a third aspect of an embodiment of the present application proposes an electronic device, which includes a memory and a processor, wherein the memory stores a computer program, and when the processor executes the computer program, the method described in the first aspect is implemented.

[0056] To achieve the above-mentioned purpose, the fourth aspect of an embodiment of the present application proposes a computer-readable storage medium, wherein the computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, the method described in the first aspect is implemented.

[0057] The speaker separation method and device, electronic device and storage medium proposed in the present application train the original voiceprint feature extraction model, the original accent feature extraction model and the original speaker separation model through sample data with multiple sample speaking objects, thereby improving the learning ability of the model, so that the voiceprints and speech features of different speakers in the target speech data can be accurately extracted, thereby achieving accurate speaker separation and improving the accuracy of speaker separation. BRIEF DESCRIPTION OF THE DRAWINGS

[0058] Figure 1 is a flow chart of a speaker separation method provided in an embodiment of the present application;

[0059] Figure 2 yes Figure 1 Flow chart of step S103 in FIG.

[0060] Figure 3 yes Figure 1 Flow chart of step S104 in FIG.

[0061] Figure 4 yes Figure 1 Flow chart of step S105 in FIG.

[0062] Figure 5 yes Figure 1 Flow chart of step S106 in FIG.

[0063] Figure 6 yes Figure 5 Flow chart of step S506 in FIG.

[0064] Figure 7 yes Figure 1 Flow chart of step S107 in FIG.

[0065] Figure 8 is a structural schematic diagram of a speaker separation device provided in an embodiment of the present application;

[0066] Fig. 9 It is a schematic diagram of the hardware structure of the electronic device provided in the embodiment of the present application. DETAILED DESCRIPTION

[0067] In order to make the purpose, technical solution and advantages of the present application more clearly understood, the present application is further described in detail below in conjunction with the accompanying drawings and embodiments. It should be understood that the specific embodiments described herein are only used to explain the present application and are not used to limit the present application.

[0068] It should be noted that, although the functional modules are divided in the device schematic diagram and the logical order is shown in the flowchart, in some cases, the steps shown or described may be performed in a different order than the module division in the device or the order in the flowchart. The terms "first", "second", etc. in the specification, claims and the above drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence.

[0069] Unless otherwise defined, all technical and scientific terms used herein have the same meaning as those commonly understood by those skilled in the art to which this application belongs. The terms used herein are only for the purpose of describing the embodiments of this application and are not intended to limit this application.

[0070] First, some nouns involved in this application are analyzed:

[0071] Artificial Intelligence (AI): AI is a comprehensive technical science that aims to develop, research, apply and maintain theories, methods, technologies and application systems that can simulate, extend and expand human intelligence. Simply put, AI is the science of enabling machines to perform complex tasks that usually require human intelligence. These tasks include but are not limited to learning (acquiring new knowledge from data), reasoning (drawing conclusions from known information), self-correction (adjusting strategies based on feedback), natural language processing (understanding and generating human language), computer vision (recognizing and understanding objects in images and videos), expert systems (mimicking the decision-making process of human experts in a specific field), etc. The research fields of AI are broad, including machine learning, deep learning, computer vision, natural language processing, robotics, expert systems, etc.

[0072] Speaker separation is an audio processing technology that aims to separate the speech of each speaker from a mixed audio signal containing multiple speakers speaking simultaneously, so that the speech signal of each speaker is independent of each other in time and space, which is convenient for subsequent speech recognition, voiceprint recognition or other audio analysis tasks.

[0073] Speaker separation technology can be applied in a variety of scenarios. For example, in the financial field, speaker separation is performed on the voice data between customers and customer service, and the customer's voice data is screened out for processing such as sentiment analysis and intent recognition.

[0074] Currently, speaker separation methods mainly extract and analyze the voiceprint features of speakers. However, when multiple speakers have similar voiceprint features, the accuracy of speaker separation cannot be guaranteed.

[0075] Based on this, the embodiments of the present application provide a speaker separation method and apparatus, an electronic device, and a storage medium, aiming to improve the accuracy of speaker separation.

[0076] The speaker separation method and device, electronic device and storage medium provided in the embodiments of the present application are specifically described through the following embodiments. First, the speaker separation method in the embodiments of the present application is described.

[0077] The embodiments of the present application can acquire and process relevant data based on artificial intelligence technology. Artificial Intelligence (AI) is the theory, method, technology and application system that uses digital computers or machines controlled by digital computers to simulate, extend and expand human intelligence, perceive the environment, acquire knowledge and use knowledge to obtain the best results.

[0078] AI basic technologies generally include sensors, dedicated AI chips, cloud computing, distributed storage, big data processing technology, operation / interaction systems, mechatronics, etc. AI software technologies mainly include computer vision technology, robotics technology, biometrics technology, speech processing technology, natural language processing technology, and machine learning / deep learning.

[0079] The speaker separation method provided in the embodiment of the present application relates to the field of artificial intelligence and financial technology. The speaker separation method provided in the embodiment of the present application can be applied to a terminal, can be applied to a server side, and can also be software running in a terminal or a server side. In some embodiments, the terminal can be a smart phone, a tablet computer, a laptop computer, a desktop computer, etc.; the server side can be configured as an independent physical server, or a server cluster or a distributed system composed of multiple physical servers, or a cloud server that provides basic cloud computing services such as cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communications, middleware services, domain name services, security services, CDN, and big data and artificial intelligence platforms; the software can be an application that implements the speaker separation method, etc., but is not limited to the above forms.

[0080] The present application can be used in many general or special computer system environments or configurations. For example: personal computers, server computers, handheld or portable devices, tablet devices, multiprocessor systems, microprocessor-based systems, set-top boxes, programmable consumer electronics, network PCs, minicomputers, mainframe computers, distributed computing environments including any of the above systems or devices, etc. The present application can be described in the general context of computer-executable instructions executed by a computer, such as program modules. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform specific tasks or implement specific abstract data types. The present application can also be practiced in distributed computing environments, in which tasks are performed by remote processing devices connected through a communication network. In a distributed computing environment, program modules can be located in local and remote computer storage media including storage devices.

[0081] It should be noted that in each specific implementation of the present application, when it comes to the need to perform relevant processing based on data related to user identity or characteristics such as user information, user behavior data, user historical data, and user location information, the user's permission or consent will be obtained first, and the collection, use, and processing of these data will comply with relevant laws, regulations, and standards. In addition, when the embodiment of the present application needs to obtain the user's sensitive personal information, the user's separate permission or consent will be obtained through a pop-up window or by jumping to a confirmation page. After clearly obtaining the user's separate permission or consent, the necessary user-related data for the normal operation of the embodiment of the present application will be obtained.

[0082] Figure 1 is an optional flow chart of the speaker separation method provided in the embodiment of the present application. Figure 1 The method may include but is not limited to steps S101 to S107.

[0083] Step S101, obtaining a speaker separation sample including at least two sample speakers; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaker to which the sample speech segment belongs;

[0084] Step S102, performing spectrum conversion on the sample speech data to obtain sample spectrum data;

[0085] Step S103, training a preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target voiceprint feature extraction model;

[0086] Step S104, training a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model;

[0087] Step S105, training a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model;

[0088] Step S106, speaker separation and labeling are performed on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker labeling information; wherein the target speech data includes at least two target speakers, and the target speaker labeling information is used to characterize the speech segment information to which each target speaker belongs in the target speech data;

[0089] Step S107 , performing speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

[0090] In steps S101 to S107 shown in the embodiment of the present application, by utilizing sample data with multiple sample speakers, the original voiceprint feature extraction model, the original accent feature extraction model, and the original speaker separation model are trained, thereby improving the learning ability of the model, being able to accurately extract the voiceprints and speech features of different speakers in the speech data, thereby achieving accurate speaker separation and improving the accuracy of speaker separation.

[0091] It should be noted that the speaker separation sample is a multi-speaker speech sample. The speaker separation sample has at least two sample speaking objects. The speaker separation sample includes sample speech data and sample speaker annotation information. The sample speech data includes multiple sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which each sample speech segment belongs.

[0092] Before step S101 in some embodiments, the speaker separation method further includes but is not limited to the following steps:

[0093] Get training speech data;

[0094] Perform data enhancement processing on the training speech data to obtain sample speech data;

[0095] Annotate the sample speech data to obtain the sample speaker annotation information;

[0096] The sample speech data and the sample speaker annotation data are combined to obtain a speaker separation sample.

[0097] Specifically, the step of "performing data enhancement processing on the training voice data to obtain sample voice data" includes obtaining voice segments of different speakers and silent / noisy voice segments, and randomly splicing multiple voice segments into the training voice data to obtain sample voice data.

[0098] Among them, random splicing is based on the following rules:

[0099] Two adjacent speech segments cannot be two silent / noisy speech segments;

[0100] Two adjacent speech segments cannot be from the same speaker;

[0101] Two adjacent speech segments cannot overlap.

[0102] In step S102 of some embodiments, the sample speech data may be spectrally converted by the Fbank method; the sample speech data may also be spectrally converted by the fast Fourier transform (FFT), short-time Fourier transform (STFT), Mel-frequency cepstral coefficient (MFCC), etc. Specifically, the selection needs to be made in combination with the actual application scenario, and is not limited thereto.

[0103] In some embodiments, the number of sample spectra is Mel spectrum data of a preset data dimension, and the preset data dimension needs to be determined according to the input parameter requirements of the original voiceprint feature extraction model, the original accent feature extraction model and the original speaker separation model, but is not limited thereto.

[0104] See also Figure 2 In some embodiments, step S103 may include but is not limited to steps S201 to S205:

[0105] Step S201, extracting voiceprint features from sample spectrum data using an original voiceprint feature extraction model to obtain original voiceprint features;

[0106] Step S202, downsampling the original voiceprint features to obtain sample voiceprint features;

[0107] Step S203, performing speaker prediction based on the sample voiceprint features to obtain first predicted speaker labeling information;

[0108] Step S204, performing loss calculation based on the sample speaker annotation information and the first predicted speaker annotation information to obtain voiceprint prediction loss data;

[0109] Step S205, optimizing the original voiceprint feature extraction model based on the voiceprint prediction loss data to obtain a target voiceprint feature extraction model.

[0110] In steps S201 to S205 shown in the embodiment of the present application, the original voiceprint features of the sample spectrum data are first extracted through the original voiceprint feature extraction model to obtain the voiceprint information of each sample speaker; the original voiceprint features are downsampled to retain the key voiceprint information and reduce the amount of calculation, the downsampled sample voiceprint features are used to predict the speaker, and the loss is calculated with the sample speaker annotation information to obtain voiceprint prediction loss data, which is used to optimize the original voiceprint feature extraction model, improve the model's recognition ability and robustness for the speaker's voiceprint, improve the accuracy of voiceprint feature extraction, and provide a solid foundation for subsequent speaker separation.

[0111] In some embodiments, the original voiceprint extraction model is a model that can extract the voiceprint features of the speaker from the spectrum data. Specifically, it can be an ECAPA-TDNN model, or a ResNetSE model, an ERes2Net model, etc., but is not limited thereto.

[0112] In some embodiments, step S202 may include but is not limited to the following steps:

[0113] Based on the preset voiceprint pooling window length, the original voiceprint features are framed to obtain multiple sample voiceprint frame features and a sample voiceprint frame extraction sequence;

[0114] The average frame features of each sample voiceprint are calculated to obtain the average frame features of multiple sample voiceprints;

[0115] Based on the sample voiceprint frame extraction order, the average frame features of multiple sample voiceprints are spliced ​​to obtain the sample voiceprint features.

[0116] It should be noted that the voiceprint pooling window length is used to compress the voiceprint features, which can reduce the size of the feature map, thereby reducing the number of parameters for subsequent processing and improving computational efficiency. Specifically, the dimension of the voiceprint pooling window length needs to be determined based on the data dimension of the original voiceprint features and the input and output of the model, but is not limited thereto.

[0117] In some embodiments, downsampling may also be achieved by performing maximum pooling processing on the original voiceprint features.

[0118] It should be noted that, since voice data is continuous, the unit of the original voiceprint feature can be frame level.

[0119] In step S203 of some embodiments, the original voiceprint features of each frame are used to perform speaker prediction through a preset first speaker predictor, thereby outputting a first predicted speaker label for each frame, and then the first predicted speaker labels of all frames are summarized to obtain first predicted speaker label information; wherein the first predicted speaker label information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0120] Specifically, the voice segment information includes the start time and the end time of the voice segment.

[0121] In one embodiment, the preset first speaker predictor is a pre-trained speaker recognition model, and the first speaker predictor can be constructed by a network structure such as AAM-Softmax, KMeans-Subcenter, Inter-TopK, etc.

[0122] In step S204 of some embodiments, loss data of each frame is obtained by performing loss calculation on the first predicted speaker annotation and sample speaker annotation information of each frame; and the loss data of each frame is summed to obtain voiceprint prediction loss data.

[0123] Among them, the spectrum loss calculation can be calculated using loss functions such as mean square error loss function, cross entropy loss function, likelihood loss function, adversarial loss function, etc., but is not limited to this.

[0124] In step S205 of some embodiments, the original voiceprint feature extraction model may be optimized based on the voiceprint prediction loss data using back propagation method, gradient descent method, momentum update method, etc., to obtain a target voiceprint feature extraction model.

[0125] See also Figure 3 In some embodiments, step S104 may include but is not limited to steps S301 to S305:

[0126] Step S301, extracting accent features from sample spectrum data using an original accent feature extraction model to obtain original accent features;

[0127] Step S302, downsampling the original accent feature to obtain a sample accent feature;

[0128] Step S303, performing speaker prediction based on the accent features of the sample to obtain second predicted speaker annotation information;

[0129] Step S304, performing loss calculation based on the sample speaker annotation information and the second predicted speaker annotation information to obtain accent prediction loss data;

[0130] Step S305 , optimizing the original accent feature extraction model based on the accent prediction loss data to obtain a target accent feature extraction model.

[0131] In steps S301 to S305 shown in the embodiment of the present application, the original accent features of the sample spectrum data are extracted by the original accent feature extraction model to obtain the accent information of each sample speaker; the original accent features are downsampled to retain the key accent information and reduce the amount of calculation; the downsampled sample accent features are used to predict the speaker, and the loss is calculated with the sample speaker annotation information to obtain accent prediction loss data, which is used to optimize the original accent feature extraction model, improve the model's recognition ability and robustness for the speaker's accent, improve the accuracy of the accent feature extraction, and provide a solid foundation for subsequent speaker separation.

[0132] In some embodiments, the original accent extraction model is a model that can extract the accent features of the speaker from spectral data. Specifically, it can be an ECAPA-TDNN model, or a CNN model, an RNN model, an LSTM model, etc., but is not limited thereto.

[0133] In some embodiments, step S302 may include but is not limited to the following steps:

[0134] Based on a preset accent pooling window length, the original accent feature is framed to obtain a plurality of sample accent frame features and a sample accent frame extraction sequence;

[0135] Calculate the average of each sample accent frame feature to obtain the average frame features of multiple sample accents;

[0136] Based on the sample accent frame extraction order, the average frame features of multiple sample accents are concatenated to obtain the sample accent features.

[0137] It should be noted that the accent pooling window length is used to compress the accent features, which can reduce the size of the feature map, thereby reducing the number of parameters for subsequent processing and improving computational efficiency. Specifically, the dimension of the accent pooling window length needs to be determined based on the data dimension of the original accent features and the input and output of the model, but is not limited thereto.

[0138] In some embodiments, downsampling may also be achieved by performing maximum pooling on the original accent features.

[0139] It should be noted that, since the speech data is continuous, the unit of the original accent feature can be frame level.

[0140] In step S303 of some embodiments, the original accent features of each frame are used to perform speaker prediction through a preset second speaker predictor, thereby outputting the second predicted speaker labeling of each frame, and then the second predicted speaker labels of all frames are summarized to obtain second predicted speaker labeling information; wherein the second predicted speaker labeling information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0141] Specifically, the voice segment information includes the start time and the end time of the voice segment.

[0142] In one embodiment, the preset second speaker predictor is a pre-trained speaker recognition model, which may be composed of a fully connected layer for classifying the speaker, and the output category is the category of the accent or dialect, so as to correspond to different speakers.

[0143] In step S304 of some embodiments, loss data of each frame is obtained by performing loss calculation on the second predicted speaker annotation and sample speaker annotation information of each frame; and the loss data of each frame is summed to obtain accent prediction loss data.

[0144] Among them, the spectrum loss calculation can be calculated using loss functions such as mean square error loss function, cross entropy loss function, likelihood loss function, adversarial loss function, etc., but is not limited to this.

[0145] In step S305 of some embodiments, the original accent feature extraction model may be optimized based on the accent prediction loss data using methods such as back propagation, gradient descent, and momentum update to obtain a target accent feature extraction model.

[0146] See also Figure 4 In some embodiments, step S105 may include but is not limited to steps S401 to S407:

[0147] Step S401, extracting speaker features from sample spectrum data using an original speaker separation model to obtain original speaker separation features;

[0148] Step S402, downsampling the original speaker separation features to obtain sample speaker separation features;

[0149] Step S403, performing feature splicing on the sample voiceprint feature, the sample accent feature and the sample speaker separation feature to obtain sample speaker separation data;

[0150] Step S404, performing attention calculation on the sample speaker separation data to obtain the sample attention feature;

[0151] Step S405, performing speaker separation on the sample attention features to obtain third predicted speaker labeling information;

[0152] Step S406, performing loss calculation based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data;

[0153] Step S407 , optimizing the original speaker separation model based on the separation prediction loss data to obtain a target speaker separation model.

[0154] In steps S401 to S407 shown in the embodiment of the present application, speaker features are extracted from sample spectrum data through an original speaker separation model to obtain original speaker separation features of each sample speaker object, the original speaker separation features include voiceprint, accent, timbre, pitch and other information of the sample speaker object, the original speaker separation features are downsampled, which not only retains key speaker information but also reduces the amount of calculation, the downsampled sample speaker separation data is feature spliced ​​with the sample voiceprint features and the sample accent features to obtain sample speaker separation data; attention calculation is performed on the sample speaker separation data to obtain sample attention features; speaker separation is performed on the sample attention features to obtain third predicted speaker annotation information; loss calculation is performed based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data, which is used to optimize the original speaker separation model, improve the model's recognition ability and robustness of speaker features, improve the accuracy of speaker feature extraction, and provide a solid foundation for subsequent speaker separation.

[0155] In some embodiments, the original voiceprint extraction model is a model that can extract speaker separation features from spectral data, specifically, it can be a Conformer model, or a Branchformer model, an EfficientConformer model, a Squeezeformer model, etc., but is not limited thereto.

[0156] In some embodiments, step S402 may include but is not limited to the following steps:

[0157] Based on the preset speaker separation pooling window length, the original speaker separation features are framed to obtain multiple sample speaker frame features and a sample speaker frame extraction order;

[0158] The frame features of each sample speaker are averaged to obtain the average frame features of multiple sample speakers;

[0159] Based on the sample speaker frame extraction order, the average frame features of multiple sample speakers are concatenated to obtain the sample speaker separation features.

[0160] It should be noted that the speaker separation pooling window length is used to compress the speaker separation features, which can reduce the size of the feature map, thereby reducing the number of parameters for subsequent processing and improving computational efficiency. Specifically, the dimension of the speaker separation pooling window length needs to be determined based on the data dimension of the original speaker separation features and the input and output of the model, but is not limited thereto.

[0161] In some embodiments, downsampling may also be achieved by performing maximum pooling on the original speaker separation features.

[0162] It should be noted that the voiceprint pooling window length, the accent pooling window length, and the speaker separation pooling window length can be set to be the same or different, and are not limited thereto.

[0163] In step S403 of some embodiments, since the sample voiceprint features, the sample accent features and the sample speaker separation features are all in vector form, the feature concatenation is to concatenate the sample voiceprint features, the sample accent features and the sample speaker separation features into a feature vector in sequence, that is:

[0164] Sample speaker separation data = [sample voiceprint feature, sample accent feature, sample speaker separation feature].

[0165] In step S404 of some embodiments, attention calculation is performed on the sample speaker separation data through a Transformer Encoder layer with a self-attention mechanism to obtain a sample attention feature.

[0166] Specifically, step S404 may include but is not limited to the following steps:

[0167] Performing matrix transformation on the sample speaker separation data to obtain a sample speaker separation transformation matrix; wherein the sample speaker separation transformation matrix includes a sample query transformation matrix, a sample key transformation matrix, and a sample value transformation matrix;

[0168] Aggregate the sample query transformation matrix and the sample key transformation matrix to obtain the sample attention score;

[0169] Normalize the sample attention score to obtain the sample attention weight;

[0170] The sample attention weight and sample value transformation matrix are aggregated and calculated to obtain the sample attention feature.

[0171] By introducing the attention mechanism to learn and capture the dependencies between different parts of the speaker separation features, and assigning different weights to different parts of the speaker separation features, the model's ability to learn speaker separation features is improved, which helps to improve the accuracy of speaker separation.

[0172] Specifically, by calculating the dot product of the sample query transformation matrix and the sample key transformation matrix, and applying the softmax function to normalize the calculation according to the column dimension (usually the sequence length dimension), the sample attention weight is obtained, which reflects the correlation strength of different parts in the speaker separation feature.

[0173] The sample attention weight and the sample value transformation matrix are weighted to obtain the sample attention feature.

[0174] In step S405 of some embodiments, speaker separation is performed on the sample attention features through a preset speaker prediction model to obtain third predicted speaker annotation information; wherein the third predicted speaker annotation information is used to characterize the speech segment information of each sample speaker in the sample speech data.

[0175] In one embodiment, the preset speaker prediction model can classify the speaker based on the sample attention features, thereby predicting different speaking objects. Specifically, the speaker prediction model can be composed of a fully connected layer. The output category is the category probability of each speaking object, thereby corresponding to different speaking objects.

[0176] In step S406 of some embodiments, separation prediction loss data is obtained by performing loss calculation on the sample speaker annotation information and the third predicted speaker annotation information;

[0177] In some embodiments, the loss calculation process includes calculating a permutation invariance loss function and a consistency correlation coefficient loss function;

[0178] By aggregating the permutation invariance loss data and the consistency correlation coefficient loss data, the separation prediction loss data is obtained.

[0179] In step S407 of some embodiments, the original speaker separation model may be optimized based on the separation prediction loss data using methods such as back propagation, gradient descent, and momentum update to obtain a target speaker separation model.

[0180] See also Figure 5 In some embodiments, step S106 may also include but is not limited to steps S501 to S506:

[0181] Step S501, performing spectrum conversion on the target speech data to obtain target speech spectrum data;

[0182] Step S502, extracting the voiceprint features of the target speech spectrum data through the target voiceprint feature extraction model to obtain the target voiceprint features;

[0183] Step S503, extracting the accent features of the target speech spectrum data using the target accent feature extraction model to obtain the target accent features;

[0184] Step S504, extracting speaker features from the target speech spectrum data using a target speaker separation model to obtain target speaker separation features;

[0185] Step S505, performing feature splicing on the target voiceprint feature, the target accent feature and the target speaker separation feature to obtain target speaker separation data;

[0186] Step S506: Speaker separation is performed based on the target speaker separation data to obtain target speaker labeling information.

[0187] In steps S501 to S506 shown in the embodiment of the present application, target speech spectrum data is obtained by performing spectrum conversion on the target speech data, and then the target speech spectrum data is processed by a target voiceprint feature extraction model, a target accent feature extraction model and a target speaker separation model respectively, and the target voiceprint features, target accent features and target speaker separation features are accurately extracted from the target speech spectrum data. The target voiceprint features, target accent features and target speaker separation features are further feature spliced ​​to obtain target speaker separation data, which integrates multi-dimensional feature information and enhances the data's ability to represent the speaker's identity. Finally, speaker separation is performed based on the target speaker separation data to obtain target speaker annotation information, thereby improving the accuracy of speaker separation.

[0188] In step S501 of some embodiments, the target speech data may be spectrally converted by the Fbank method; the target speech data may also be spectrally converted by the fast Fourier transform (FFT), short-time Fourier transform (STFT), Mel-frequency cepstral coefficient (MFCC) and other methods. Specifically, the selection needs to be made in combination with the actual application scenario, and is not limited thereto.

[0189] In some embodiments, the number of sample spectra is Mel-spectrogram data of a preset data dimension, and the preset data dimension needs to be determined according to the input parameter requirements of the target voiceprint feature extraction model, the target accent feature extraction model, and the target speaker separation model, but is not limited thereto.

[0190] In some embodiments, step S502 includes but is not limited to the following steps:

[0191] The target voiceprint feature extraction model is used to extract the voiceprint features of the target speech spectrum data to obtain the initial voiceprint features;

[0192] The initial voiceprint features are downsampled to obtain the target voiceprint features.

[0193] Specifically, the step of “downsampling the initial voiceprint features to obtain the target voiceprint features” may include but is not limited to the following steps:

[0194] Based on a preset voiceprint pooling window length, the initial voiceprint feature is framed to obtain a plurality of initial voiceprint frame features and an initial voiceprint frame extraction sequence;

[0195] Calculate the average of each initial voiceprint frame feature to obtain multiple initial voiceprint average frame features;

[0196] Based on the initial voiceprint frame extraction order, multiple initial voiceprint average frame features are concatenated to obtain the target voiceprint feature.

[0197] In some embodiments, downsampling may also be achieved by performing maximum pooling processing on the initial voiceprint features.

[0198] In some embodiments, step S503 includes but is not limited to the following steps:

[0199] The target accent feature extraction model is used to extract the accent features of the target speech spectrum data to obtain the initial accent features;

[0200] The initial accent features are downsampled to obtain the target accent features.

[0201] Specifically, the step of “downsampling the initial accent feature to obtain the target accent feature” may include but is not limited to the following steps:

[0202] Based on a preset accent pooling window length, the initial accent feature is framed to obtain a plurality of initial accent frame features and an initial accent frame extraction sequence;

[0203] Calculate the average of each initial accent frame feature to obtain multiple initial accent average frame features;

[0204] Based on the initial accent frame extraction order, multiple initial accent average frame features are concatenated to obtain the target accent feature.

[0205] In some embodiments, downsampling may also be achieved by performing a maximum pooling process on the initial accent features.

[0206] In some embodiments, step S504 may include but is not limited to the following steps:

[0207] The target speaker separation model is used to extract speaker features from the target speech spectrum data to obtain initial speaker separation features;

[0208] The initial speaker separation features are downsampled to obtain the target speaker separation features.

[0209] Specifically, the step of “downsampling the initial speaker separation features to obtain target speaker separation features” may include but is not limited to the following steps:

[0210] Based on a preset speaker separation pooling window length, the initial speaker separation feature is framed to obtain a plurality of initial speaker frame features and an initial speaker frame extraction sequence;

[0211] Calculate the average of each initial speaker's frame features to obtain multiple initial speaker average frame features;

[0212] Based on the initial speaker frame extraction order, the average frame features of multiple initial speakers are concatenated to obtain the target speaker separation features.

[0213] In some embodiments, downsampling may also be achieved by performing maximum pooling on the initial speaker separation features.

[0214] In step S505 of some embodiments, since the target voiceprint feature, the target accent feature and the target speaker separation feature are all in vector form, the feature concatenation is to concatenate the target voiceprint feature, the target accent feature and the target speaker separation feature into a feature vector in sequence, that is:

[0215] Target speaker separation data = [target voiceprint features, target accent features, target speaker separation features].

[0216] See also Figure 6 In some embodiments, step S506 includes but is not limited to steps S601 to S602:

[0217] Step S601, performing attention calculation on the target speaker separation data to obtain a target attention feature;

[0218] Step S602: Speaker separation is performed on the target attention feature to obtain target speaker labeling information.

[0219] Steps S601 to S602 shown in the embodiment of the present application can focus on key feature areas during speaker separation by performing attention calculation on the target speaker separation data. Therefore, speaker separation based on the target attention features can make the speaker separation result more consistent with the actual context and the characteristics of the speaker, thereby improving the accuracy of speaker separation.

[0220] In some embodiments, step S601 may include but is not limited to the following steps:

[0221] Performing matrix transformation on the target speaker separation data to obtain a target speaker separation transformation matrix; wherein the target speaker separation transformation matrix includes a target query transformation matrix, a target key transformation matrix, and a target value transformation matrix;

[0222] Aggregate the target query transformation matrix and the target key transformation matrix to obtain the target attention score;

[0223] Normalize the target attention score to obtain the target attention weight;

[0224] The target attention weight and target value transformation matrix are aggregated to obtain the target attention feature.

[0225] It should be noted that the specific implementation of the above step S601 is the same as the implementation shown in step S404, and will not be repeated here.

[0226] See also Figure 7 In some embodiments, step S107 may include but is not limited to steps S701 to S702:

[0227] Step S701, for each target speaker, obtaining the speaker's speech period according to the target speaker's annotation information;

[0228] Step S702: performing speech extraction on the target speech data based on the speaker speech period to obtain the target speaker speech data of each target speaker.

[0229] In steps S701 to S702 shown in the embodiment of the present application, the target speaker annotation information is parsed to obtain the speech period of each target speaker in the target speech data, thereby ensuring that the speech segment corresponding to each target speaker can be accurately extracted from the entire target speech data, thereby improving the accuracy of speaker separation.

[0230] It should be noted that, since the target speech data contains at least two target speakers, the target speaker annotation information is used to characterize the speech segment information of each target speaker in the target speech data; correspondingly, the speech segment information is the speaker speech period of each target speaker, including the start time and end time of the speech segment.

[0231] In step S702 of some embodiments, target speaker speech data of each target speaker is extracted from the target speech data according to the speaker speech period (start time and end time of the speech segment), thereby completing speaker separation.

[0232] See also Figure 8 The present application also provides a speaker separation device, which can implement the above-mentioned speaker separation method, and the device includes:

[0233] The sample data acquisition module 801 is used to acquire a speaker separation sample including at least two sample speakers; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaker to which the sample speech segment belongs;

[0234] The sample spectrum conversion module 802 is used to perform spectrum conversion on the sample speech data to obtain sample spectrum data;

[0235] The voiceprint feature extraction model training module 803 is used to perform model training on a preset original voiceprint feature extraction model based on sample spectrum data and sample speaker annotation information to obtain a target voiceprint feature extraction model;

[0236] The accent feature extraction model training module 804 is used to perform model training on a preset original accent feature extraction model based on sample spectrum data and sample speaker annotation information to obtain a target accent feature extraction model;

[0237] The speaker separation model training module 805 is used to perform model training on a preset original speaker separation model based on sample spectrum data and sample speaker annotation information to obtain a target speaker separation model;

[0238] The target speaker annotation module 806 is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein the target speech data includes at least two target speakers, and the target speaker annotation information is used to characterize the speech segment information to which each target speaker belongs in the target speech data;

[0239] The speaker speech separation module 807 is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

[0240] The specific implementation of the speaker separation device is substantially the same as the specific implementation of the above-mentioned speaker separation method, and will not be described in detail herein.

[0241] The embodiment of the present application also provides an electronic device, the electronic device includes a memory and a processor, the memory stores a computer program, and the processor implements the above-mentioned speaker separation method when executing the computer program. The electronic device can be any intelligent terminal including a tablet computer, a car computer, etc.

[0242] See also Fig. 9 , Fig. 9 The hardware structure of an electronic device of another embodiment is illustrated, and the electronic device includes:

[0243] The processor 901 may be implemented by a general-purpose CPU (Central Processing Unit), a microprocessor, an application-specific integrated circuit (ASIC), or one or more integrated circuits, and is used to execute relevant programs to implement the technical solutions provided in the embodiments of the present application;

[0244] The memory 902 can be implemented in the form of a read-only memory (ROM), a static storage device, a dynamic storage device, or a random access memory (RAM). The memory 902 can store an operating system and other application programs. When the technical solution provided in the embodiment of this specification is implemented by software or firmware, the relevant program code is stored in the memory 902, and the processor 901 calls and executes the speaker separation method of the embodiment of this application;

[0245] Input / output interface 903, used to implement information input and output;

[0246] Communication interface 904, used to realize communication interaction between the device and other devices, which can be realized through wired mode (such as USB, network cable, etc.) or wireless mode (such as mobile network, WI FI, Bluetooth, etc.);

[0247] A bus 905 that transmits information between the various components of the device (e.g., the processor 901, the memory 902, the input / output interface 903, and the communication interface 904);

[0248] The processor 901 , the memory 902 , the input / output interface 903 and the communication interface 904 are connected to each other in communication within the device via a bus 905 .

[0249] The embodiment of the present application further provides a computer-readable storage medium, which stores a computer program. When the computer program is executed by a processor, the above-mentioned speaker separation method is implemented.

[0250] The memory, as a non-transient computer-readable storage medium, can be used to store non-transient software programs and non-transient computer executable programs. In addition, the memory may include a high-speed random access memory, and may also include a non-transient memory, such as at least one disk storage device, a flash memory device, or other non-transient solid-state storage device. In some embodiments, the memory may optionally include a memory remotely disposed relative to the processor, and these remote memories may be connected to the processor via a network. Examples of the above-mentioned network include, but are not limited to, the Internet, an intranet, a local area network, a mobile communication network, and combinations thereof.

[0251] The speaker separation method and device, electronic device and storage medium provided in the embodiments of the present application train the original voiceprint feature extraction model, the original accent feature extraction model and the original speaker separation model through sample data with multiple sample speaking objects, thereby improving the learning ability of the model, so that the voiceprints and speech features of different speakers in the target speech data can be accurately extracted, thereby achieving accurate speaker separation and improving the accuracy of speaker separation.

[0252] The embodiments described in the embodiments of the present application are intended to more clearly illustrate the technical solutions of the embodiments of the present application and do not constitute a limitation on the technical solutions provided in the embodiments of the present application. Those skilled in the art will appreciate that with the evolution of technology and the emergence of new application scenarios, the technical solutions provided in the embodiments of the present application are also applicable to similar technical problems.

[0253] Those skilled in the art will appreciate that the technical solutions shown in the figures do not constitute a limitation on the embodiments of the present application, and may include more or fewer steps than shown in the figures, or a combination of certain steps, or different steps.

[0254] The device embodiments described above are merely illustrative, and the units described as separate components may or may not be physically separated, that is, they may be located in one place or distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0255] Those skilled in the art will appreciate that all or some of the steps in the methods disclosed above, and the functional modules / units in the systems and devices may be implemented as software, firmware, hardware, or a suitable combination thereof.

[0256] The terms "first", "second", "third", "fourth", etc. (if any) in the specification of the present application and the above-mentioned drawings are used to distinguish similar objects, and are not necessarily used to describe a specific order or sequence. It should be understood that the data used in this way can be interchangeable where appropriate, so that the embodiments of the present application described herein can be implemented in an order other than those illustrated or described herein. In addition, the terms "including" and "having" and any of their variations are intended to cover non-exclusive inclusions, for example, a process, method, system, product or device comprising a series of steps or units is not necessarily limited to those steps or units clearly listed, but may include other steps or units that are not clearly listed or inherent to these processes, methods, products or devices.

[0257] It should be understood that in the present application, "at least one (item)" means one or more, and "plurality" means two or more. "And / or" is used to describe the association relationship of associated objects, indicating that three relationships may exist. For example, "A and / or B" can mean: only A exists, only B exists, and A and B exist at the same time, where A and B can be singular or plural. The character " / " generally indicates that the objects associated before and after are in an "or" relationship. "At least one of the following" or similar expressions refers to any combination of these items, including any combination of single or plural items. For example, at least one of a, b or c can mean: a, b, c, "a and b", "a and c", "b and c", or "a and b and c", where a, b, c can be single or multiple.

[0258] In the several embodiments provided in the present application, it should be understood that the disclosed devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the above units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be through some interfaces, indirect coupling or communication connection of devices or units, which can be electrical, mechanical or other forms.

[0259] The units described above as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0260] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0261] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including multiple instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods of various embodiments of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (Read-Only Memory, referred to as ROM), random access memory (Random Access Memory, referred to as RAM), disk or optical disk and other media that can store programs.

[0262] The preferred embodiments of the present invention are described above with reference to the accompanying drawings, but the scope of the rights of the present invention is not limited thereto. Any modification, equivalent substitution and improvement made by a person skilled in the art without departing from the scope and essence of the present invention should be within the scope of the rights of the present invention.

Claims

1. A speaker separation method, characterized in that: The method comprises: Acquire a speaker separation sample including at least two sample speaking objects; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs; Performing spectrum conversion on the sample speech data to obtain sample spectrum data; Based on the sample spectrum data and the sample speaker annotation information, a preset original voiceprint feature extraction model is trained to obtain a target voiceprint feature extraction model; Based on the sample spectrum data and the sample speaker annotation information, a preset original accent feature extraction model is trained to obtain a target accent feature extraction model; Based on the sample spectrum data and the sample speaker annotation information, a preset original speaker separation model is trained to obtain a target speaker separation model; The target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model are used to perform speaker separation and labeling on the pre-acquired target speech data to obtain target speaker labeling information; wherein the target speech data includes at least two target speakers, and the target speaker labeling information is used to characterize the speech segment information to which each target speaker belongs in the target speech data; The target speech data is subjected to speech separation based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

2. The method according to claim 1, characterized in that The method of training a preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target voiceprint feature extraction model includes: Extracting voiceprint features from the sample spectrum data using the original voiceprint feature extraction model to obtain original voiceprint features; Downsampling the original voiceprint features to obtain sample voiceprint features; Perform speaker prediction based on the sample voiceprint feature to obtain first predicted speaker labeling information; Performing loss calculation based on the sample speaker annotation information and the first predicted speaker annotation information to obtain voiceprint prediction loss data; The original voiceprint feature extraction model is optimized based on the voiceprint prediction loss data to obtain the target voiceprint feature extraction model.

3. The method according to claim 2, characterized in that The method of training a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model includes: Extracting accent features from the sample spectrum data using the original accent feature extraction model to obtain original accent features; Downsampling the original accent feature to obtain a sample accent feature; Perform speaker prediction based on the sample accent feature to obtain second predicted speaker annotation information; Performing loss calculation based on the sample speaker annotation information and the second predicted speaker annotation information to obtain accent prediction loss data; The original accent feature extraction model is optimized based on the accent prediction loss data to obtain the target accent feature extraction model.

4. The method according to claim 3, characterized in that The method of training a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model includes: Extracting speaker features from the sample spectrum data using the original speaker separation model to obtain original speaker separation features; Downsampling the original speaker separation feature to obtain a sample speaker separation feature; Performing feature splicing on the sample voiceprint feature, the sample accent feature and the sample speaker separation feature to obtain sample speaker separation data; Performing attention calculation on the sample speaker separation data to obtain a sample attention feature; Performing speaker separation on the sample attention features to obtain third predicted speaker labeling information; Perform loss calculation based on the sample speaker annotation information and the third predicted speaker annotation information to obtain separation prediction loss data; The original speaker separation model is optimized based on the separation prediction loss data to obtain the target speaker separation model.

5. The method according to any one of claims 1 to 4, characterized in that: The speaker separation and labeling of the pre-acquired target speech data by using the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker labeling information includes: Performing spectrum conversion on the target speech data to obtain target speech spectrum data; Performing voiceprint feature extraction on the target speech spectrum data using the target voiceprint feature extraction model to obtain target voiceprint features; Extracting the target accent feature from the target speech spectrum data using the target accent feature extraction model to obtain a target accent feature; Extracting speaker features from the target speech spectrum data using the target speaker separation model to obtain target speaker separation features; Performing feature splicing on the target voiceprint feature, the target accent feature and the target speaker separation feature to obtain target speaker separation data; Speaker separation is performed based on the target speaker separation data to obtain target speaker labeling information.

6. The method according to claim 5, characterized in that The performing speaker separation based on the target speaker separation data to obtain target speaker labeling information includes: Performing attention calculation on the target speaker separation data to obtain a target attention feature; Speaker separation is performed on the target attention feature to obtain the target speaker labeling information.

7. The method according to any one of claims 1 to 4, characterized in that: The performing speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker includes: For each target speaker, obtaining the speaker's speech period according to the target speaker's annotation information; The target voice data is subjected to voice extraction based on the speaker voice period to obtain the target speaker voice data of each target speaker.

8. A speaker separation device, characterized in that: The device comprises: A sample data acquisition module, used to acquire a speaker separation sample containing at least two sample speaking objects; wherein the speaker separation sample includes sample speech data and sample speaker annotation information, the sample speech data includes sample speech segments, and the sample speaker annotation information is used to characterize the sample speaking object to which the sample speech segment belongs; A sample spectrum conversion module, used for performing spectrum conversion on the sample speech data to obtain sample spectrum data; A voiceprint feature extraction model training module is used to perform model training on a preset original voiceprint feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target voiceprint feature extraction model; An accent feature extraction model training module, used for training a preset original accent feature extraction model based on the sample spectrum data and the sample speaker annotation information to obtain a target accent feature extraction model; A speaker separation model training module, used for training a preset original speaker separation model based on the sample spectrum data and the sample speaker annotation information to obtain a target speaker separation model; A target speaker annotation module is used to perform speaker separation annotation on the pre-acquired target speech data through the target voiceprint feature extraction model, the target accent feature extraction model and the target speaker separation model to obtain target speaker annotation information; wherein the target speech data contains at least two target speakers, and the target speaker annotation information is used to characterize the speech segment information to which each target speaker belongs in the target speech data; The speaker speech separation module is used to perform speech separation on the target speech data based on the target speaker annotation information to obtain the target speaker speech data of each target speaker.

9. An electronic device, characterized in that: The electronic device comprises a memory and a processor, the memory stores a computer program, and the processor implements the method according to any one of claims 1 to 7 when executing the computer program.

10. A computer-readable storage medium storing a computer program, characterized in that: When the computer program is executed by a processor, the method according to any one of claims 1 to 7 is implemented.

Images