Speaker diarization method, apparatus, device, and medium

Abstract

The application relates to a speaker separation method, device, equipment and medium, the method comprising: acquiring multi-channel conversational voice collected by a recording device, wherein the recording device is arranged on a wearer; identifying the multi-channel conversational voice through a plurality of preset models to obtain a plurality of identification results, wherein the plurality of preset models comprise a sound source speaker separation model, a voiceprint speaker separation model and a semantic speaker separation model; and fusing the plurality of identification results to obtain a speaker separation result. The application sets the recording device on the wearer, then identifies the recording data in the recording device based on sound source, voiceprint and semantic information through the model, thereby determining the wearer and the non-wearer, and solving the problem that a speaker role cannot be accurately identified in a complex environment.

Description

Technical Field

[0001] This application relates to the field of deep learning technology, and in particular to a speaker separation method, apparatus, device and medium. Background Technology

[0002] With the deepening application of voice technology in various fields, the convenience and commercial value brought by intelligent voice have been continuously validated. As audio data from offline staff services is widely recorded and preserved, how to fully utilize this offline data has become a research hotspot. For example, in addition to basic speech recognition technology, to obtain accurate marketing dialogue records, it is also necessary to accurately separate the voices of employees from those of others. Voiceprint information is typically used for clustering to obtain speaker segmentation information; however, voiceprint clustering alone can only separate different speakers, not identify speaker roles, and its recognition performance is poor in situations where the number of speakers is uncertain or in noisy environments.

[0003] There is currently no effective solution to the above problems. Summary of the Invention

[0004] This application provides a speaker separation method, apparatus, device, and medium to solve the aforementioned technical problem of being unable to accurately identify speaker roles in complex environments.

[0005] According to one aspect of the embodiments of this application, this application provides a speaker separation method, including: acquiring multi-channel conversational speech collected by a recording device, wherein the recording device is installed on a wearer; recognizing the multi-channel conversational speech through multiple preset models to obtain multiple recognition results, wherein the multiple preset models include a sound source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model; and fusing the multiple recognition results to obtain a speaker separation result.

[0006] Optionally, multiple recognition results are obtained by recognizing multi-channel conversational speech through multiple preset models, including: extracting first speech and second speech from multi-channel conversational speech, wherein the first speech is the wearer's speech after noise removal, and the second speech is the speech of the wearer and non-wearer conversing in any channel; and recognizing the first speech and second speech through multiple preset models to obtain multiple recognition results.

[0007] Optionally, extracting the first speech from the multi-channel conversational speech includes: if the multi-channel conversational speech is a three-channel speech, then obtaining the first clean speech signal of the first channel and the second channel and the second clean speech signal of the first channel and the third channel; and taking the speech signal with the smallest amplitude among the first clean speech signal and the second clean speech signal as the first speech.

[0008] Optionally, obtaining the first clean speech signal of the first channel and the second channel includes: performing null beamforming on the speech of the first channel and the second channel in the direction of the connection to obtain a noise signal, wherein the direction of the connection is the direction from the first channel to the second channel; performing first-order differential beamforming on the speech of the first channel and the second channel in the opposite direction of the connection to obtain a speech signal including a small amount of noise; and subtracting the noise signal from the speech signal to obtain the first clean speech signal.

[0009] Optionally, the first speech and the second speech are recognized using multiple preset models to obtain multiple recognition results, including: recognizing the first speech using a source-speaker separation model to obtain a first recognition result; recognizing the second speech using a voiceprint-speaker separation model to obtain a second recognition result, wherein the voiceprint-speaker separation model is trained based on the average voiceprint of the wearer and non-wearer; and recognizing the second speech using a semantic speaker separation model to obtain a third recognition result.

[0010] Optionally, fusing multiple recognition results to obtain speaker separation results includes: extracting the probability distribution from multiple recognition results, wherein the probability distribution represents the probability that each frame of speech belongs to a wearer or a non-wearer; calibrating and fusing the probability distribution based on prosodic features to obtain a fused target probability distribution; and obtaining speaker separation results based on the target probability distribution, wherein the multiple recognition results include multiple frames of speech and speech tags corresponding to each frame of speech, and the speech tags include wearer tags and non-wearer tags.

[0011] Optionally, after obtaining the speaker separation result, the method further includes: obtaining the wearer's third voice from the first recognition result; comparing the third voice with the fourth voice in the speaker separation result whose voice label is not the wearer's label to obtain the voice similarity; if the similarity is higher than a preset threshold, then changing the voice label of the fourth voice to the wearer's label.

[0012] According to another aspect of the embodiments of this application, this application also provides a speaker separation device, including: an acquisition module, used to acquire multi-channel conversational speech collected by a recording device, wherein the recording device is mounted on a wearer; an identification module, used to identify the multi-channel conversational speech through multiple preset models to obtain multiple identification results, wherein the multiple preset models include a sound source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model; and a fusion module, used to fuse the multiple identification results to obtain a speaker separation result.

[0013] According to another aspect of the embodiments of this application, this application also provides an electronic device, including a memory, a processor, a communication interface and a communication bus. The memory stores a computer program that can run on the processor. The memory and the processor communicate with each other through the communication bus and the communication interface. When the processor executes the computer program, it implements the steps of any of the above methods.

[0014] According to another aspect of the embodiments of this application, this application also provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform any of the methods described above.

[0015] The technical solution of this application can be applied to the design of speech recognition and processing using deep learning technology.

[0016] Compared with related technologies, the technical solutions provided in this application have the following advantages:

[0017] This application provides a speaker separation method, comprising: acquiring multi-channel conversational speech collected by a recording device, wherein the recording device is worn by a wearer; recognizing the multi-channel conversational speech through multiple preset models to obtain multiple recognition results, wherein the multiple preset models include a sound source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model; and fusing the multiple recognition results to obtain a speaker separation result. This application solves the problem of accurately identifying speaker roles in complex environments by placing the recording device on the wearer and then identifying the recording data in the recording device based on sound source, voiceprint, and semantic information through models. Attached Figure Description

[0018] The accompanying drawings, which are incorporated in and form part of this specification, illustrate embodiments consistent with this application and, together with the description, serve to explain the principles of this application.

[0019] To more clearly illustrate the technical solutions in the embodiments of this application or related technologies, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, those skilled in the art can obtain other drawings based on these drawings without creative effort.

[0020] Figure 1 This is a flowchart of an optional speaker separation method provided according to an embodiment of this application;

[0021] Figure 2 This is a schematic diagram of an optional voiceprint classification model structure provided according to an embodiment of this application;

[0022] Figure 3A flowchart illustrating an optional clustering method provided according to an embodiment of this application;

[0023] Figure 4 A flowchart illustrating an optional decision-making method provided according to an embodiment of this application;

[0024] Figure 5 This is a schematic diagram of an optional semantic speaker separation model provided according to an embodiment of this application;

[0025] Figure 6 This is a schematic diagram of an optional dual-microphone voice beamforming method according to an embodiment of this application;

[0026] Figure 7 This is a schematic diagram of an optional three-microphone voice beamforming according to an embodiment of this application;

[0027] Figure 8 This is a schematic diagram of an optional speaker separation system provided according to an embodiment of this application;

[0028] Figure 9 This is a block diagram of an optional speaker separation device provided according to an embodiment of this application;

[0029] Figure 10 This is a schematic diagram of an optional electronic device structure provided according to an embodiment of this application. Detailed Implementation

[0030] To make the objectives, technical solutions, and advantages of the embodiments of this application clearer, the technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0031] In the following description, the use of suffixes such as "module," "part," or "unit" to denote elements is solely for the purpose of illustration and has no specific meaning in itself. Therefore, "module" and "part" may be used interchangeably.

[0032] With the deepening application of voice technology in various fields, the convenience and commercial value brought by intelligent voice have been continuously validated. As audio data from offline staff services is widely recorded and documented, how to fully utilize this offline data has become a research hotspot. Collected recordings can help companies improve employees' marketing skills, uncover user needs, and reflect employee work status. Besides basic voice recognition technology, this requires accurately separating the voices of employees from those of others to obtain accurate marketing dialogue records. This necessitates identifying the identities of salespeople and other individuals during the marketing process. Current mainstream technologies typically use voiceprint clustering to obtain speaker segmentation information. However, voiceprint clustering solutions usually only separate different speakers and cannot determine which role is a salesperson, and their effectiveness is poor in situations where the number of speakers is uncertain or in noisy environments. In real-world scenarios, the data collected by the device often includes multiple speakers, and in different physical environments, the speakers may have different positions (standing opposite each other, parallel to the left, parallel to the right, standing in front of or behind each other, etc.), different distances (about 1 meter, about 2 meters, about 3 meters, etc.), and different venues (in a standalone store, shopping mall, inside a motor vehicle, etc.). These factors drive the need to integrate a variety of differentiated technical means to achieve speaker separation.

[0033] To address the problems mentioned in the background art, according to one aspect of an embodiment of this application, this application provides a speaker separation method, such as... Figure 1 As shown, it includes:

[0034] Step 101: Acquire multi-channel conversational voice recorded by a recording device, wherein the recording device is mounted on the wearer.

[0035] Step 103: Recognize multi-channel conversational speech through multiple preset models to obtain multiple recognition results. Among them, the multiple preset models include a source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model.

[0036] Step 105: Merge multiple recognition results to obtain the speaker separation result.

[0037] Current mainstream voiceprint-based clustering methods have several drawbacks when applied to marketing scenarios: First, they cannot assign sales or customer identities to the clustered individuals; second, they are ineffective with multiple speakers when the number of speakers is unknown, and perform poorly in noisy environments. This application addresses these issues by integrating acoustic front-end processing algorithms, voiceprint, and semantic information to clearly identify the wearer while improving the accuracy of speaker separation in complex offline scenarios.

[0038] The embodiments provided in this application are applied in a marketing scenario, where the recording device is placed on the wearer (i.e., the salesperson). Therefore, the speaker separation result obtained by this application is whether each frame of speech belongs to the wearer (i.e., the salesperson) or the non-wearer (i.e., the customer). In other words, this application can accurately distinguish between salespersons and customers in a marketing scenario.

[0039] As an optional embodiment, multiple recognition results are obtained by recognizing multi-channel conversational speech through multiple preset models, including: extracting a first speech and a second speech from the multi-channel conversational speech, wherein the first speech is the wearer's speech after noise removal, and the second speech is the speech of the wearer and non-wearer conversing, collected from any channel; and recognizing the first speech and the second speech through multiple preset models to obtain multiple recognition results.

[0040] Specifically, after processing by the front-end differential array algorithm, most of the non-wearer's speech is suppressed, while the first speech (i.e., the wearer's speech) is preserved. Since this application adopts multi-channel recording, the conversational speech collected from one of the channels is selected as the second speech. The second speech includes the wearer's speech, the non-wearer's speech, and some noise.

[0041] This application provides wearer voice enhancement technology based on front-end beamforming.

[0042] Several preset models are included, such as the voice source speaker separation model, the voiceprint speaker separation model, and the semantic speaker separation model. These models will be explained below.

[0043] Regarding the speaker-source separation model: On the hardware side, based on the speaker's positional differences, the sound source information from the microphone array of the recording device is fused to achieve a separation model based on the direction information of the sound source. Therefore, this application uses a speech activity detection model to extract the sound source information processed by the front-end differential array algorithm and perform speaker separation.

[0044] Alternatively, the source information extraction in the source-speaker separation model may not use a speech detection model, but instead use a differential algorithm to directly determine the probability of suppression or other features.

[0045] Regarding the speaker separation model using voiceprints, this application provides the following solutions, including:

[0046] (1) End-to-end voiceprint classification model: The differences in position and volume between wearers and non-wearers make their speech distinguishable. By improving the existing neural network-based end-to-end speaker separation model, the average voiceprints of wearers and non-wearers are trained, enabling the model to distinguish between wearers and non-wearers. Figure 2The diagram shows the structure of the end-to-end voiceprint classification model provided in this application. The model uses self-attention as the convolutional layer of the neural network. The improved speaker separation model labels the voiceprinter and non-voicer in the labeled data instead of simply assigning roles A and B. Furthermore, the model no longer uses permutation-free loss but instead uses BCE loss (Binary cross-entropy loss) as the loss function.

[0047] This application provides an improved end-to-end speaker classification model structure.

[0048] Alternatively, the end-to-end speaker classification model can also use other general neural network architectures such as LSTM as the encoder.

[0049] (2) Use the front-end algorithm to extract the voiceprint of the current wearer and obtain the character cluster through the clustering algorithm. Figure 3 The flowchart of the clustering method provided in this application is shown in the figure. Simultaneously, the clustered role information corresponding to the actual wearer is obtained by calculating the similarity between each cluster and the wearer's voiceprint, or the overlap with the separation result of the voice source speaker. The wearer's voiceprint can be obtained through manual registration and / or automatic filtering using algorithms.

[0050] (3) Using the Personal VAD (Voice Activity Detection) method. Figure 4 The flowchart of the decision-making method provided in this application is shown in the figure. The voiceprint of the current wearer is extracted using a front-end algorithm or manually registered, and the speaker separation result is determined.

[0051] Regarding the semantic speaker separation model: This considers the distinct difference between the speech of the wearer (salesperson) and the non-wearer (customer). For example, "Welcome" is a standard wearer's speech, while "How much is this?" is a standard non-wearer's speech. Figure 5 The following is a schematic diagram of a semantic speaker separation model proposed in this application. As shown in the figure, after the full speech is recognized by ASR (Automatic Speech Recognition), the probability distribution of each word for the wearer and non-wearer is obtained by the BERT model.

[0052] Optionally, BERT in the semantic speaker separation model can be replaced with other neural language models.

[0053] As an optional embodiment, extracting the first speech from multi-channel conversational speech includes: if the multi-channel conversational speech is three-channel speech, then obtaining the first clean speech signal of the first channel and the second channel and the second clean speech signal of the first channel and the third channel; and taking the speech signal with the smallest amplitude among the first clean speech signal and the second clean speech signal as the first speech.

[0054] As an optional embodiment, obtaining the first clean speech signal of the first channel and the second channel includes: performing null beamforming on the speech of the first channel and the second channel in the direction of connection to obtain a noise signal, wherein the direction of connection is the direction from the first channel to the second channel; performing first-order differential beamforming on the speech of the first channel and the second channel in the opposite direction of connection to obtain a speech signal including a small amount of noise; and subtracting the noise signal from the speech signal to obtain the first clean speech signal.

[0055] The multi-channel voice technology in this application includes, but is not limited to, dual-microphone and triple-microphone technologies. Figure 6 This is a schematic diagram of dual-microphone speech beamforming provided in this application. In the diagram, "1" represents the microphone of the first channel, "2" represents the microphone of the second channel, "zero-notch beamforming" means that the speech signals of the first and second channels are subjected to zero-notch beamforming along the line connecting the first and second channels to obtain a noise signal; "first-order differential beamforming" means that the speech signals of the first and second channels are subjected to first-order differential beamforming in the opposite direction of the line connecting them to obtain a speech signal with a small amount of noise; "subtraction" means that the speech signal with a small amount of noise obtained by first-order differential beamforming is subtracted from the noise signal obtained by zero-notch beamforming to obtain a relatively clean speech signal Y12.

[0056] Figure 7 This is a schematic diagram of the three-microphone speech beamforming provided in this application, where "1" is the microphone of the first channel, "2" is the microphone of the second channel, "3" is the microphone of the third channel, "Mic1Mic2 beamforming" means obtaining a relatively clean first speech signal from the first and second channels (the formation method is the same as the dual-microphone speech beamforming method described above); "Mic1Mic3 beamforming" means obtaining a relatively clean second speech signal from the first and third channels, and "min" means performing minimum value processing on the two relatively clean speech signals obtained to obtain and output a cleaner speech signal.

[0057] This application provides a method for acquiring speech on a recording device using a microphone array with a specific arrangement. A small smart badge or recorder facilitates audio data collection from service personnel in offline service and sales scenarios. The collected audio data undergoes anti-interference and noise reduction processing, resulting in accurate and clean audio data from the wearer. For dual-microphone speech, a differential array algorithm is used to obtain the desired direction signal and null beam estimation noise signal, and then the desired target signal is obtained. To further achieve better beamforming, a multi-microphone speech combination scheme is used to narrow the beamwidth and more accurately collect signals from specific sound sources.

[0058] As an optional embodiment, multiple recognition results are obtained by recognizing the first speech and the second speech through multiple preset models, including: recognizing the first speech through a source-speaker separation model to obtain a first recognition result; recognizing the second speech through a voiceprint-speaker separation model to obtain a second recognition result, wherein the voiceprint-speaker separation model is trained based on the average voiceprint of the wearer and non-wearer; and recognizing the second speech through a semantic speaker separation model to obtain a third recognition result.

[0059] The average voiceprint of a non-wearer can be understood as the voiceprint of a non-wearer collected by taking into account the factor of distance.

[0060] As an optional embodiment, fusing multiple recognition results to obtain speaker separation results includes: extracting the probability distribution from the multiple recognition results, wherein the probability distribution represents the probability that each frame of speech belongs to a wearer or a non-wearer; calibrating and fusing the probability distribution based on prosodic features to obtain a fused target probability distribution; and obtaining speaker separation results based on the target probability distribution, wherein the multiple recognition results include multiple frames of speech and speech tags corresponding to each frame of speech, and the speech tags include wearer tags and non-wearer tags.

[0061] Specifically, the three preset models mentioned above can output the probability distribution of wearers and non-wearers in each frame, and can also output their own speaker separation results. For the frame-level probability distribution, this application combines the prosodic features of speech (such as energy, fundamental frequency, etc.) to calibrate and fuse the probability distribution, outputting the fused probability distribution for each frame. After smoothing, the speaker separation results fused by the model are output.

[0062] This application also provides a scheme that can use an additional model to fuse, calibrate, and decode the speaker separation probability at the frame level.

[0063] Optionally, the fusion calibration in the fusion model can be a traditional machine learning model (logistic regression, decision tree, etc.) or a neural network-based classification model.

[0064] As an optional embodiment, after obtaining the speaker separation result, the method further includes: obtaining the wearer's third voice from the first recognition result; comparing the third voice with the fourth voice in the speaker separation result whose voice label is not the wearer's label to obtain the voice similarity; if the similarity is higher than a preset threshold, then changing the voice label of the fourth voice to the wearer's label.

[0065] Specifically, the separation results after model fusion have achieved high accuracy. However, considering that both end-to-end speaker separation models and semantic speaker separation models require a large amount of data for training and suffer from generalization issues, this application proposes to use speaker verification to correct the fusion results from a natural person's perspective using voiceprints. Speaker verification uses the wearer's speech or manually registered audio with high confidence in the fused separation results as the registered speech, and compares it with the segments output by the fusion model. If the similarity is high but the fusion model judges it as a non-wearer, the speaker in that segment is changed to the wearer, and vice versa. The voiceprint model uses the ResNet structure and the AAM-softmax loss function.

[0066] Regarding the wearer's voiceprint, this application can use an algorithm to automatically select the obtained audio, and can also accept manually registered audio from the device.

[0067] Based on the speaker separation method, this application also provides a speaker separation system, such as... Figure 8 As shown, the system comprises three parts: front-end, speaker separation, and model fusion. The front-end includes acquiring multi-channel recordings, obtaining clean wearer speech based on the front-end algorithm, and acquiring full speech from any channel of the multi-channel recordings. The full speech includes conversations between the wearer and non-wearer. The speaker separation part includes inputting the wearer's speech into the source speaker separation model for recognition, inputting the full speech into the voiceprint speaker separation model and the semantic speaker separation model respectively, and then inputting the outputs of the three models into the model fusion part for fusion. The model fusion part includes fusing the three recognition results, and then performing voiceprint comparison on the fused separation results. The wearer's voiceprint used for voiceprint comparison is extracted in advance from the output of the source speaker separation model.

[0068] This application comprehensively considers sound source information, voiceprint information, and semantic information, and through the combination of hardware and software, it can significantly improve the speaker separation accuracy in sales scenarios. Table 1 shows the results of the speaker separation system in this application on the business test set. The performance evaluation uses the industry-standard DER (diarization error rate) as the evaluation metric; the smaller the DER, the better the speaker separation effect. As can be seen from the table, the DER decreased by 41.4% after model fusion.

[0069] Table 1

[0070] Voice source and speaker separation 13.3％ 25.9％ 18.9％ This system 5.4％ 12.5％ 16.6％

[0071] This application provides a speaker separation method, comprising: acquiring multi-channel conversational speech collected by a recording device, wherein the recording device is worn by a wearer; recognizing the multi-channel conversational speech using multiple preset models to obtain multiple recognition results, wherein the multiple preset models include a sound source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model; and fusing the multiple recognition results to obtain a speaker separation result. This application solves the problem of accurately identifying speaker roles in complex environments by placing the recording device on the wearer and then using models to identify the recording data in the recording device based on sound source, voiceprint, and semantic information, thereby determining the wearer and non-wearer.

[0072] According to another aspect of the embodiments of this application, this application also provides a speaker separation device, such as... Figure 9 As shown, it includes:

[0073] The acquisition module 902 is used to acquire multi-channel conversational voice collected by the recording device, wherein the recording device is installed on the wearer;

[0074] The recognition module 904 is used to recognize multi-channel conversational speech through multiple preset models and obtain multiple recognition results. Among them, the multiple preset models include a source speaker separation model, a voiceprint speaker separation model and a semantic speaker separation model.

[0075] The fusion module 906 is used to fuse multiple recognition results to obtain the speaker separation result.

[0076] It should be noted that the acquisition module 902 in this embodiment can be used to execute step 101 in this application embodiment, the identification module 904 in this embodiment can be used to execute step 103 in this application embodiment, and the fusion module 906 in this embodiment can be used to execute step 105 in this application embodiment.

[0077] Optionally, the identification module 904 includes:

[0078] The extraction submodule is used to extract the first speech and the second speech from the multi-channel conversation speech. The first speech is the wearer's speech after noise removal, and the second speech is the speech of the wearer and non-wearer in conversation collected from any channel.

[0079] The recognition submodule is used to recognize the first and second speech using multiple preset models to obtain multiple recognition results.

[0080] Optionally, the extraction submodule includes:

[0081] The acquisition unit is used to acquire the first clean speech signal of the first channel and the second channel and the second clean speech signal of the first channel and the third channel if the multi-channel conversation speech is a three-channel speech.

[0082] The determining unit is used to select the speech signal with the smallest amplitude between the first pure speech signal and the second pure speech signal as the first speech signal.

[0083] Optionally, the acquisition unit is further configured to perform null beamforming on the speech of the first channel and the second channel in the connecting direction to obtain a noise signal, wherein the connecting direction is the direction from the first channel to the second channel; perform first-order differential beamforming on the speech of the first channel and the second channel in the opposite direction of the connecting direction to obtain a speech signal including a small amount of noise; and subtract the noise signal from the speech signal to obtain a first clean speech signal.

[0084] Optionally, the recognition submodule is further configured to recognize the first speech using a voice source-speaker separation model to obtain a first recognition result; recognize the second speech using a voiceprint-speaker separation model to obtain a second recognition result, wherein the voiceprint-speaker separation model is trained based on the average voiceprint of the wearer and non-wearer; and recognize the second speech using a semantic speaker separation model to obtain a third recognition result.

[0085] Optionally, the fusion module 906 is further used to extract the probability distribution from multiple recognition results, wherein the probability distribution represents the probability that each frame of speech belongs to the wearer or non-wearer; calibrate and fuse the probability distribution based on prosodic features to obtain the fused target probability distribution; and obtain the speaker separation result based on the target probability distribution, wherein the multiple recognition results include multiple frames of speech and speech tags corresponding to each frame of speech, and the speech tags include wearer tags and non-wearer tags.

[0086] Optionally, the fusion module 906 is further configured to, after obtaining the speaker separation result, obtain the wearer's third voice from the first recognition result; compare the third voice with the fourth voice in the speaker separation result whose voice label is not the wearer's label, and obtain the voice similarity; if the similarity is higher than a preset threshold, change the voice label of the fourth voice to the wearer's label.

[0087] It should be noted that the examples and application scenarios implemented by the above modules and corresponding steps are the same, but are not limited to the content disclosed in the above embodiments.

[0088] According to another aspect of the embodiments of this application, such as Figure 10 As shown, this application provides an electronic device, including a memory 1001, a processor 1002, a communication interface 1003, and a communication bus 1004. The memory 1001 stores a computer program that can run on the processor 1002. The memory 1001 and the processor 1002 communicate through the communication bus 1004 and the communication interface 1003. When the processor 1002 executes the computer program, it implements the steps of the above method.

[0089] The memory and processor in the aforementioned electronic devices communicate with each other via a communication bus and a communication interface. The communication bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. This communication bus can be divided into an address bus, a data bus, a control bus, etc.

[0090] The memory may include random access memory (RAM) or non-volatile memory, such as at least one disk storage device. Optionally, the memory may also be at least one storage device located remotely from the aforementioned processor.

[0091] The processors mentioned above can be general-purpose processors, including central processing units (CPUs), network processors (NPs), etc.; they can also be digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0092] According to another aspect of the embodiments of this application, this application provides a computer-readable medium having processor-executable non-volatile program code that causes the processor to perform the steps of any of the methods described above.

[0093] Optionally, in embodiments of this application, the computer-readable medium is configured to store program code for the processor to perform the above-described method steps.

[0094] Optionally, specific examples in this embodiment can refer to the examples described in the above embodiments, and will not be repeated here.

[0095] In specific implementation, the embodiments of this application can be referred to the above embodiments and have corresponding technical effects.

[0096] It is understood that the embodiments described herein can be implemented in hardware, software, firmware, middleware, microcode, or a combination thereof. For hardware implementation, the processing unit can be implemented in one or more application-specific integrated circuits (ASICs), digital signal processors (DSPs), digital signal processing devices (DSPDs), programmable logic devices (PLDs), field-programmable gate arrays (FPGAs), general-purpose processors, controllers, microcontrollers, microprocessors, other electronic units for performing the functions described herein, or combinations thereof.

[0097] For software implementation, the techniques described herein can be implemented by units that perform the functions described herein. The software code can be stored in memory and executed by a processor. The memory can be implemented in the processor or external to the processor.

[0098] Those skilled in the art will recognize that the units and algorithm steps of the various examples described in conjunction with the embodiments disclosed herein can be implemented in electronic hardware, or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0099] Those skilled in the art will understand that, for the sake of convenience and brevity, the specific working processes of the systems, devices, and units described above can be referred to the corresponding processes in the foregoing method embodiments, and will not be repeated here.

[0100] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely illustrative. For instance, the division of modules is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple modules or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interfaces, devices, or units, and may be electrical, mechanical, or other forms.

[0101] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0102] In addition, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit.

[0103] If the aforementioned function is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the embodiments of this application, or the part that contributes to the prior art, or a part of the technical solution, can be embodied in the form of a software product. This computer software product is stored in a storage medium and includes several instructions to cause a computer device (which may be a personal computer, server, or network device, etc.) to execute all or part of the steps of the methods described in the various embodiments of this application. The aforementioned storage medium includes various media capable of storing program code, such as USB flash drives, portable hard drives, ROM, RAM, magnetic disks, or optical disks. It should be noted that in this document, relational terms such as "first" and "second" are used only to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Furthermore, the terms "comprising," "including," or any other variations thereof are intended to cover non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a process, method, article, or apparatus. In the absence of further restrictions, an element defined by the phrase "comprising a..." does not exclude the presence of other identical elements in the process, method, article, or apparatus that includes said element.

[0104] The above description is merely a specific embodiment of this application, enabling those skilled in the art to understand or implement this application. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the general principles defined herein may be implemented in other embodiments without departing from the spirit or scope of this application. Therefore, this application is not to be limited to the embodiments shown herein, but is to be accorded the widest scope consistent with the principles and novel features claimed herein.

Claims

1. A speaker diarization method, characterized by, include: Acquire multi-channel conversational voice recorded by a recording device, wherein the recording device is mounted on the wearer; The multi-channel conversational speech is identified by multiple preset models, resulting in multiple recognition results. The multiple preset models include a source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model. By fusing the multiple identification results, the speaker separation result is obtained; The step of recognizing the multi-channel conversational speech using multiple preset models to obtain multiple recognition results includes: extracting a first speech and a second speech from the multi-channel conversational speech, wherein the first speech is the wearer's speech after noise removal, and the second speech is the speech of the wearer and a non-wearer conversing, collected from any channel; and recognizing the first speech and the second speech using multiple preset models to obtain the multiple recognition results. The step of recognizing the first speech and the second speech using multiple preset models to obtain the multiple recognition results includes: recognizing the first speech using the source-speaker separation model to obtain a first recognition result; recognizing the second speech using the voiceprint-speaker separation model to obtain a second recognition result, wherein the voiceprint-speaker separation model is trained based on the average voiceprint of the wearer and the non-wearer; and recognizing the second speech using the semantic-speaker separation model to obtain a third recognition result.

2. The method of claim 1, wherein, The step of extracting the first speech from the multi-channel conversational speech includes: If the multi-channel conversational speech is a three-channel speech, then the first clean speech signal of the first channel and the second channel and the second clean speech signal of the first channel and the third channel are obtained; The speech signal with the smallest amplitude between the first clean speech signal and the second clean speech signal is taken as the first speech signal.

3. The method of claim 2, wherein, The acquisition of the first clean speech signal from the first channel and the second channel includes: Null beamforming is performed on the speech signals of the first channel and the second channel in the direction of connection to obtain a noise signal, wherein the direction of connection is the direction from the first channel to the second channel; First-order differential beamforming is performed on the speech signals of the first channel and the second channel in the opposite direction of the connecting line to obtain a speech signal including a small amount of noise. The first clean speech signal is obtained by subtracting the noise signal from the speech signal.

4. The method according to claim 1, characterized in that, The fusion of the multiple recognition results to obtain the speaker separation result includes: Extract the probability distribution from the multiple recognition results, wherein the probability distribution represents the probability that each frame of speech belongs to the wearer or the non-wearer; The probability distribution is calibrated and fused based on prosodic features to obtain the fused target probability distribution. Based on the target probability distribution, the speaker separation result is obtained, wherein the multiple recognition results include multiple frames of speech and speech tags corresponding to each frame of speech, and the speech tags include wearer tags and non-wearer tags.

5. The method according to claim 4, characterized in that, After obtaining the speaker separation result, the method further includes: The wearer's third voice is obtained from the first recognition result; The third speech is compared with the fourth speech in the speaker separation result whose speech label is the non-wearer label to obtain the speech similarity; If the similarity is higher than a preset threshold, the voice tag of the fourth voice is changed to the wearer's tag.

6. A speaker separation device, characterized in that, include: The acquisition module is used to acquire multi-channel conversational voice collected by the recording device, wherein the recording device is installed on the wearer. The recognition module is used to recognize the multi-channel conversational speech through multiple preset models and obtain multiple recognition results. The multiple preset models include a source speaker separation model, a voiceprint speaker separation model, and a semantic speaker separation model. The fusion module is used to fuse the multiple recognition results to obtain the speaker separation result; The identification module includes: An extraction submodule is used to extract a first voice and a second voice from the multi-channel conversational voice, wherein the first voice is the voice of the wearer after noise removal, and the second voice is the voice of the wearer talking to a non-wearer collected from any channel; The recognition submodule is used to recognize the first speech and the second speech through multiple preset models to obtain the multiple recognition results; The recognition submodule is further configured to recognize the first speech using the source-speaker separation model to obtain a first recognition result; recognize the second speech using the voiceprint-speaker separation model to obtain a second recognition result, wherein the voiceprint-speaker separation model is trained based on the average voiceprint of the wearer and the non-wearer; and recognize the second speech using the semantic speaker separation model to obtain a third recognition result.

7. An electronic device comprising a memory, a processor, a communication interface, and a communication bus, wherein the memory stores a computer program executable on the processor, and the memory and the processor communicate via the communication bus and the communication interface, characterized in that... When the processor executes the computer program, it implements the steps of the method described in any one of claims 1 to 5.

8. A computer-readable medium having processor-executable non-volatile program code, characterized in that, The program code causes the processor to execute the method of any one of claims 1 to 5.

Images