A method, device, electronic device and storage medium for separating talkers
By vocal determination and feature extraction of target audio, combined with clustering technology, real-time online speaker separation in multiple speaker scenarios is achieved, the problems of real-time and cost in the existing technology are solved, and the separation efficiency is improved and effects.
Patent Information
- Application Number
- CN202210203569.2
- Authority / Receiving Office
- CN · China
- Patent Type
- Patents(China)
- Current Assignee / Owner
- Filing Date
- 2022-03-03
- Publication Date
- 2024-11-08
- Estimated Expiration
- 2042-03-03
AI Technical Summary
The prior art is difficult to achieve real-time online speaker separation in multiple speaker scenarios, and requires a large amount of labeled data for training, resulting in high cost and low efficiency.
By vocal determination and feature extraction of the target audio, clustering based on feature similarity, first coarse clustering and then fine clustering, real-time and fast separation of users is achieved without a large amount of labeled data.
Real-time online speaker separation in multiple speaker scenarios is achieved, reducing training costs and time, and improving separation effect.
Smart Images

Figure CN114596877B_ABST
Abstract
Description
Technical Field
[0001] The present disclosure relates to the field of computer technology, in particular to the field of voice processing and intelligent customer service technology, and specifically to a method, device, electronic device and storage medium for separating speakers. Background Art
[0002] In scenarios such as intelligent customer service, conference discussions, interviews, and variety shows, there are usually multiple speakers on a single channel, so speaker separation is required. Summary of the invention
[0003] The present disclosure provides a talker separation method, apparatus, device and storage medium.
[0004] According to one aspect of the present disclosure, a method for separating speakers is provided, comprising:
[0005] Performing vocal determination on the first audio segment of the target audio to obtain multiple vocal segments;
[0006] Extracting features from the vocal segments to obtain speaker features of each vocal segment;
[0007] Based on the feature similarity between the speaker features of each vocal segment and a preset first similarity threshold, the vocal segments are clustered to obtain at least one speaker segment set and at least one non-speaker segment set; wherein the number of vocal segments included in each speaker segment set reaches a preset number and belongs to the same speaker; the number of vocal segments included in the non-speaker segment set does not reach the preset number;
[0008] For each vocal segment included in the non-speaker segment set, the feature similarity between the vocal segment and each speaker segment set is calculated, and the maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, the vocal segment is divided into the speaker segment set corresponding to the maximum feature similarity.
[0009] According to one aspect of the present disclosure, there is provided a speaker separation device, comprising:
[0010] A determination module, configured to perform vocal determination on a first audio segment of the target audio to obtain a plurality of vocal segments;
[0011] A first feature extraction module, used to extract features from the vocal segments to obtain speaker features of each vocal segment;
[0012] A clustering module, configured to cluster the vocal segments based on the feature similarity between the speaker features of the respective vocal segments and a preset first similarity threshold, to obtain at least one speaker segment set and at least one non-speaker segment set; wherein each speaker segment set includes vocal segments of a preset number and belongs to the same speaker; and the non-speaker segment set includes vocal segments of a preset number or less;
[0013] A calculation module is used to calculate the feature similarity between each vocal segment contained in the non-speaker segment set and each speaker segment set, and determine the maximum feature similarity therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, the vocal segment is divided into the speaker segment set corresponding to the maximum feature similarity.
[0014] According to another aspect of the present disclosure, there is provided an electronic device, comprising:
[0015] at least one processor; and
[0016] a memory communicatively connected to the at least one processor; wherein,
[0017] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a talker separation method.
[0018] According to another aspect of the present disclosure, a non-transitory computer-readable storage medium storing computer instructions is provided, wherein the computer instructions are used to cause the computer to execute a talker separation method.
[0019] According to yet another aspect of the present disclosure, a computer program product is provided, including a computer program, which implements a talker separation method when executed by a processor.
[0020] By applying the speaker separation method provided by the embodiment of the present disclosure, the first audio segment of the target audio is first subjected to vocal determination to obtain a vocal segment, and then the speech features of the vocal segment are extracted. Subsequently, the first clustering, i.e., coarse clustering, is performed based on the feature similarity between the speaker features of the vocal segments, and a cluster set that reaches a preset number is determined as a speaker segment set. The coarse clustering has a low clustering delay and ensures the high purity of each speaker segment set. Then, based on the feature similarity between the vocal segments contained in the non-speaker segment set and each speaker segment set, the vocal segments in the non-speaker segment set are re-divided, thereby achieving further fine clustering, thereby performing real-time and detailed adjustments to the coarse clustering results to obtain real-time and fast speaker separation results.
[0021] Since the feature extraction and clustering processes can be performed online, and the clustering results can include multiple speaker segment sets, the speaker separation method provided by the embodiment of the present disclosure is suitable for online speaker separation of online audio containing multiple speakers, and has a good speaker separation effect even when there are many speakers. In addition, a large amount of labeled data for speaker separation is not required, and the audio samples for training the feature extraction network only need to be labeled as a whole, which can be obtained from public data sets, and the cost is low.
[0022] It should be understood that the content described in this section is not intended to identify the key or important features of the embodiments of the present disclosure, nor is it intended to limit the scope of the present disclosure. Other features of the present disclosure will become easily understood through the following description. BRIEF DESCRIPTION OF THE DRAWINGS
[0023] The accompanying drawings are used to better understand the present solution and do not constitute a limitation of the present disclosure.
[0024] Figure 1 A schematic diagram of a speaker separation method for online audio in the related art;
[0025] Figure 2 A flow chart of a method for separating speakers provided by an embodiment of the present disclosure;
[0026] Figure 3(a) is a schematic diagram of an undirected graph before pruning;
[0027] Figure 3(b) is a schematic diagram of an undirected graph after pruning;
[0028] Figure 4 A schematic diagram of a process of tag matching provided in an embodiment of the present disclosure;
[0029] Figure 5 A schematic diagram of a speaker separation method provided by an embodiment of the present disclosure;
[0030] Figure 6 is a block diagram of a device for implementing the speaker separation method of the embodiment of the present disclosure;
[0031] Figure 7 The present invention is a block diagram of an electronic device for implementing the speaker separation method according to an embodiment of the present disclosure. DETAILED DESCRIPTION
[0032] The following is a description of exemplary embodiments of the present disclosure in conjunction with the accompanying drawings, including various details of the embodiments of the present disclosure to facilitate understanding, which should be considered as merely exemplary. Therefore, it should be recognized by those of ordinary skill in the art that various changes and modifications may be made to the embodiments described herein without departing from the scope and spirit of the present disclosure. Similarly, for the sake of clarity and conciseness, the description of well-known functions and structures is omitted in the following description.
[0033] In scenarios such as intelligent customer service, conference discussions, interviews, and variety shows, there are usually multiple speakers on a single channel, so speaker separation is required.
[0034] Usually, the speaker separation process is completed offline, that is, the complete audio needs to be obtained and the number of speakers in the audio needs to be determined before separation can be performed. However, speaker separation completed offline cannot be applied in situations where real-time requirements are high.
[0035] Currently, there is also a solution for online audio talker separation, which uses end-to-end deep learning. For details, please refer to Figure 1 , Figure 1 A schematic diagram of a speaker separation method for online audio in the related art is shown in FIG. Figure 1 As shown in the figure, it includes an audio framing module, an encoder-decoder module based on a self-attention mechanism, and a prediction module. The audio framing module frames the online audio, and then passes it through the encoder-decoder module to obtain audio features. The speaker label of each frame is directly predicted based on the audio features. The entire model is trained in an end-to-end manner.
[0036] This end-to-end processing solution has ideal processing results when the number of speakers is less than 2, but the effect decreases significantly when there are more than two speakers, that is, it is not suitable for scenarios with multiple speakers. In addition, a large amount of speaker-separated labeled data is required for training. In other words, for audio samples, it is necessary to determine who spoke each segment, that is, it is necessary to determine the speaker label of each audio segment in the sample audio data before it can be used for training. However, the labor cost of obtaining a large amount of speaker-separated labeled data is high.
[0037] In order to solve the above technical problems, the present disclosure provides a method, device, electronic device and storage medium for speaker separation.
[0038] In one embodiment of the present disclosure, a method for separating speakers is provided, the method comprising:
[0039] Performing vocal determination on the first audio segment of the target audio to obtain multiple vocal segments;
[0040] Extracting features from the vocal segments to obtain speaker features of each vocal segment;
[0041] Based on the feature similarity between the speaker features of each vocal segment and a preset first similarity threshold, the vocal segments are clustered to obtain at least one speaker segment set and at least one non-speaker segment set; wherein the number of vocal segments included in each speaker segment set reaches a preset number and belongs to the same speaker; the number of vocal segments included in the non-speaker segment set does not reach the preset number;
[0042] For each vocal segment included in the non-speaker segment set, the feature similarity between the vocal segment and each speaker segment set is calculated, and the maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, the vocal segment is divided into the speaker segment set corresponding to the maximum feature similarity.
[0043] By applying the speaker separation method provided by the embodiment of the present disclosure, the first audio segment of the target audio is first subjected to vocal determination to obtain a vocal segment, and then the speech features of the vocal segment are extracted. Subsequently, the first clustering, i.e., coarse clustering, is performed based on the feature similarity between the speaker features of the vocal segments, and a cluster set that reaches a preset number is determined as a speaker segment set. The coarse clustering has a low clustering delay and ensures the high purity of each speaker segment set. Then, based on the feature similarity between the vocal segments contained in the non-speaker segment set and each speaker segment set, the vocal segments in the non-speaker segment set are re-divided, thereby achieving further fine clustering, thereby performing real-time and detailed adjustments to the coarse clustering results to obtain real-time and fast speaker separation results.
[0044] Since the feature extraction and clustering processes can be performed online, and the clustering results can include multiple speaker segment sets, the speaker separation method provided by the embodiment of the present disclosure is suitable for online speaker separation of online audio containing multiple speakers, and has a good speaker separation effect even when there are many speakers. In addition, a large amount of labeled data for speaker separation is not required, and the audio samples for training the feature extraction network only need to be labeled as a whole, which can be obtained from public data sets, and the cost is low.
[0045] The speaker separation method, device, electronic device and storage medium provided in the embodiments of the present disclosure are respectively introduced in detail below.
[0046] See also Figure 2 , Figure 2 A flow chart of a method for separating speakers provided in an embodiment of the present disclosure is shown in FIG. Figure 2 As shown, the method may include the following steps:
[0047] S201: Performing vocal determination on a first audio segment of target audio to obtain a plurality of vocal segments.
[0048] The speaker separation method provided by the embodiments of the present disclosure can be applied to electronic devices and can perform speaker separation on online audio. That is, the electronic device can obtain online audio in real time and perform real-time speaker separation on the obtained audio segments, rather than performing speaker separation after obtaining the complete audio.
[0049] Therefore, the target audio may be online audio. In addition, the speaker separation method provided in the embodiment of the present disclosure is also applicable to offline audio, so the target audio may also be offline audio.
[0050] In this step, human voice determination is performed on the first audio segment of the target audio, wherein the first audio segment may be a currently acquired segment. The purpose of human voice determination is to determine which audio segments are human voice audio and which audio segments are non-human voice audio, and only retain human voice audio. That is, noise segments and silent segments are screened out to obtain human voice audio segments, wherein the noise may be a ring tone, a busy tone, music, an environment, etc.
[0051] Specifically, the first audio segment of the target audio may be divided into frames, each frame being 25 ms and the frame shift being 10 ms, and human voice determination may be performed on each frame.
[0052] For example, human voice sample data and non-human voice sample data such as ringback tone, busy tone, music, etc. obtained from open source data are used in advance to train GMM (Gaussian Mixed Model), DNN (Deep Neural Networks) or other speech detection models. The trained speech detection model can be used as a classifier to identify whether an audio frame is a human voice audio frame.
[0053] In addition, whether the audio frame is a silent audio frame can be identified by the short-time energy of the audio frame, that is, if the short-time energy of the audio frame is 0, it indicates that the audio frame is a silent audio frame.
[0054] For the first audio segment of the target audio, the noise segment and the silent segment are screened out to obtain the vocal segment. The length of each vocal segment can be set according to the requirements, for example, to about 1.5 seconds.
[0055] S202: Extract features from the vocal segments to obtain speaker features of each vocal segment.
[0056] In the disclosed embodiment, a pre-trained feature extraction network can be used to extract speaker features of each vocal segment. The feature extraction network can be a pre-trained deep learning network model, and the structure of the deep learning network model can sequentially include: multi-layer LSTM (long-short term memory) or TDNN (time delay neural network), Linear mapping layer, FC (full connection, full connection layer).
[0057] The feature extraction network can be trained in advance using open source datasets such as Aishell (a large-scale public Chinese speech database) or LibriSpeech (a large-scale public English speech database). Different from the related art, in the embodiments of the present disclosure, no speaker-separated annotated data is required. In other words, for audio samples, it is not necessary to determine who said each segment, but only to determine who said the entire audio sample, which can be used for training.
[0058] As an example, a large number of audio samples are obtained from the Aishell open source dataset. These audio samples are all from 100 speakers. The audio samples are divided into frames. First, common features of each frame are extracted, such as MFCC (Mel-Frequency Cepstral Coefficients) features or PLP (Perceptual Linear Predictive), Fbank (Filter Bank, frequency domain features), etc., which are usually set to 40 dimensions. Then, after passing through multiple layers of LSTM and Linear mapping layers, the speaker's voice spectrum features are obtained, which are usually set to 128 dimensions and are obtained by accumulating and averaging multiple frames. For example, the average is calculated every 150 frames as the speaker feature of a human voice segment of about 1.5s. Then, after passing through the fully connected layer and the output layer, a 100-dimensional output vector is obtained. The loss value can be calculated according to the speaker label of the audio sample, for example, the loss value is calculated using the CE (cross entropy) loss function, and then the parameters in the network are adjusted.
[0059] It can be seen that in the embodiments of the present disclosure, a large amount of speaker-separated labeled data is not required for training. The audio samples for training the feature extraction network only need to be labeled as a whole and can be obtained from public data sets, which is relatively low in cost.
[0060] After the feature extraction network is trained, it can be used to extract speaker features of vocal segments.
[0061] As an example, 150 adjacent human voice audio frames constitute a human voice segment with a duration of about 1.5 seconds, and the speaker feature of the human voice segment may be 128 dimensions.
[0062] S203: Based on the feature similarity between the speaker features of each vocal segment and a preset first similarity threshold, the vocal segments are clustered to obtain at least one speaker segment set and at least one non-speaker segment set; wherein the number of vocal segments included in each speaker segment set reaches a preset number and belongs to the same speaker; the number of vocal segments included in the non-speaker segment set does not reach the preset number.
[0063] In the disclosed embodiment, since the speaker features of the human voice segment are the speaker's spectral features, clustering can be performed based on the feature similarity of the speaker features. The feature similarity can be the inverse of the distance of the speaker features in the feature space, that is, the smaller the distance in the feature space, the higher the feature similarity.
[0064] In the embodiment of the present disclosure, fast clustering, namely coarse clustering, is first performed based on a preset first similarity threshold. The coarse clustering algorithm may be an incrementally updated bottom-up hierarchical clustering (AHC).
[0065] Specifically, coarse clustering is performed based on the feature similarity between the speaker features of each vocal segment. In order to ensure that the speaker segments clustered together belong to the same speaker, a larger first similarity threshold can be preset. As an example, the first similarity threshold can be set to 95%. In other words, only speaker segments with very high similarity are clustered into one category.
[0066] Furthermore, when the number of human voices included in a cluster set reaches a preset number, the cluster set can be considered as a speaker segment set, that is, the human voice segments in this set are considered to belong to the same speaker.
[0067] As an example, the preset number may be set to 6.
[0068] That is to say, for cluster sets that do not reach the preset number, it is not certain whether they belong to a set of vocal segments of one person. In addition, they may be abnormal clusters, such as interjections, extended sounds, overlapping sounds, or noise that has not been removed. Therefore, subsequent fine division is required.
[0069] In the disclosed embodiment, the above-mentioned coarse clustering process can ensure the purity of each cluster set, and the clustering speed is fast and the delay is small, and there is no need to wait for complete data before proceeding like traditional clustering.
[0070] The disadvantage of the above coarse clustering process is that the number of cluster sets formed is greater than the actual number of speakers, and secondary clustering, namely fine clustering, is required.
[0071] S204: For each vocal segment included in the non-speaker segment set, calculate the feature similarity between the vocal segment and each speaker segment set, and determine the maximum feature similarity therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, divide the vocal segment into the speaker segment set corresponding to the maximum feature similarity.
[0072] For the vocal segments in the non-speaker segment set generated after the coarse clustering, fine clustering needs to be performed.
[0073] First, for each vocal segment included in the non-speaker segment set, the feature similarity between the vocal segment and each speaker segment set is calculated.
[0074] In one embodiment of the present disclosure, for a certain vocal segment included in the non-speaker segment set, the similarity between the vocal segment and each vocal segment in the speaker segment set is calculated respectively, and the average value is taken as the feature similarity between the vocal segment and the speaker segment set.
[0075] For example, the non-speaker segment set A includes the vocal segment a1, and the speaker segment sets are represented by B, C and D. The feature similarities of the vocal segment a1 and the vocal segments b1, b2…bm in the speaker segment set B are calculated respectively, and the average is taken to obtain the feature similarity between the vocal segment a1 and the speaker segment set B.
[0076] It is easy to understand that the higher the feature similarity, the more likely it is that the vocal segment and the speaker segment set belong to the same speaker; if the calculated feature similarities are very small, it means that there is no set similar to the vocal segment.
[0077] Therefore, in the embodiment of the present disclosure, it is determined whether the calculated maximum feature similarity is greater than the preset second similarity threshold. If so, the vocal segment is divided into a speaker segment set corresponding to the maximum feature similarity. The preset second similarity threshold is usually smaller than the similarity threshold used in coarse clustering, i.e., the first similarity threshold. However, in order to ensure the accuracy of speaker separation, the second similarity threshold also needs to be large enough. As an example, the second similarity threshold can be set to 85%.
[0078] Continuing with the above example, if the feature similarity between vocal segment a1 and speaker segment set B is M1, the feature similarity between vocal segment a1 and speaker segment set C is M2, and the feature similarity between vocal segment a1 and speaker segment set D is M3, if the maximum feature similarity is M2 and is greater than the preset second similarity threshold, then vocal segment a1 is divided into the speaker segment set corresponding to M2, that is, speaker segment set D. If the maximum feature similarity is not greater than the preset second similarity threshold, then vocal segment a1 is not re-divided.
[0079] Other vocal segments a2, a3, etc. included in the non-speaker segment set A may be re-divided in the same manner as a1.
[0080] At this point, the vocal segments in each speaker segment set correspond to the same speaker, and a speaker label can be generated for each speaker segment set, that is, a speaker separation result for the first audio segment of the target audio is obtained.
[0081] By applying the speaker separation method provided by the embodiment of the present disclosure, the first audio segment of the target audio is first subjected to vocal determination to obtain a vocal segment, and then the speech features of the vocal segment are extracted. Subsequently, the first clustering, i.e., coarse clustering, is performed based on the feature similarity between the speaker features of the vocal segments, and a cluster set that reaches a preset number is determined as a speaker segment set. The coarse clustering has a low clustering delay and ensures the high purity of each speaker segment set. Then, based on the feature similarity between the vocal segments contained in the non-speaker segment set and each speaker segment set, the vocal segments in the non-speaker segment set are re-divided, thereby achieving further fine clustering, thereby performing real-time and detailed adjustments to the coarse clustering results to obtain real-time and fast speaker separation results.
[0082] Since the feature extraction and clustering processes can be performed online, and the clustering results can include multiple speaker segment sets, the speaker separation method provided by the embodiment of the present disclosure is suitable for online speaker separation of online audio containing multiple speakers, and has a good speaker separation effect even when there are many speakers. In addition, a large amount of labeled data for speaker separation is not required, and the audio samples for training the feature extraction network only need to be labeled as a whole, which can be obtained from public data sets, and the cost is low.
[0083] In one embodiment of the present disclosure, in the process of performing fine clustering based on the coarse clustering result, an undirected graph may be constructed, and the fine clustering may be performed based on the undirected graph.
[0084] Specifically, the method further includes: constructing an undirected graph based on the feature similarity between the speaker features of each vocal segment, the nodes of the undirected graph are the vocal segments, and the connection weights between the nodes are the feature similarities between the vocal segments.
[0085] If any two vocal segments are connected, an undirected graph with very high complexity will be generated. In order to trim the undirected graph and reduce the amount of calculation, the feature similarity between the connected nodes in the undirected graph can be set to be greater than a preset threshold. In other words, only two nodes whose feature similarity is greater than the preset similarity threshold are connected.
[0086] Referring to FIG. 3( a ) and FIG. 3( b ), FIG. 3( a ) is a schematic diagram of an undirected graph before pruning, and FIG. 3( b ) is a schematic diagram of an undirected graph after pruning.
[0087] It can be seen that before cropping, any two vocal segments are connected, which is very complex; after cropping, a large number of connection relationships are deleted, which can greatly reduce the amount of calculation in the subsequent undirected graph-based calculation process.
[0088] Accordingly, for each vocal segment included in the non-speaker segment set, calculating the feature similarity between the vocal segment and each speaker segment set, and determining the maximum feature similarity therefrom, may specifically include:
[0089] Step 11: For each vocal segment included in the non-speaker segment set, based on the number of connections between the vocal segment and the vocal segments in each speaker segment set, a speaker segment set having a connection number greater than a preset number is screened out as a candidate speaker segment set for the vocal segment.
[0090] Step 12: For each vocal segment included in the non-speaker segment set, calculate the mean similarity between the vocal segment and the vocal segments in each candidate speaker segment set, and use them as the feature similarities between the vocal segment and each candidate speaker segment set, and determine the maximum feature similarity from the feature similarities between the vocal segment and each candidate speaker segment set.
[0091] Specifically, in order to reduce the amount of calculation, preliminary screening may be performed based on the number of connections between the vocal segment and the vocal segments in each speaker segment set.
[0092] As an example, the threshold of the number of connections is set to N. For the vocal segment a included in the non-speaker segment set, the number of connections between the vocal segment a and the vocal segments in the speaker segment set B is determined. If it is greater than N, the speaker segment set B is determined as the candidate speaker segment set, that is, the speaker segment set B participates in the subsequent feature similarity calculation; otherwise, it means that the feature similarities between the vocal segment a and the vocal segments in the speaker segment set B are not high, and the vocal segment a must not belong to the speaker segment set B, that is, there is no need to perform subsequent feature similarity calculation, thereby reducing the amount of calculation.
[0093] When calculating feature similarity, the mean similarity between the vocal segments in the non-speaker segment set and the vocal segments in each candidate speaker segment set is calculated as the feature similarity between the vocal segment and each candidate speaker segment set. The specific calculation process can be found in S204 and will not be described in detail.
[0094] It can be seen that in the disclosed embodiment, a speaker feature undirected graph is constructed based on the feature similarity between the speaker features of each vocal segment, and in the speaker feature undirected graph, only two nodes with feature similarity greater than a preset similarity threshold are connected, thereby greatly reducing the complexity of the undirected graph. Furthermore, according to the number of connections between each vocal segment contained in the non-speaker segment set and the vocal segments in each speaker segment set, preliminary screening is performed to determine the candidate speaker segment set. In the subsequent fine clustering process, only the candidate speaker segment set is considered, which further reduces the computational complexity and thereby improves the real-time performance of online speaker separation.
[0095] In the disclosed embodiment, when the target audio to be processed is online audio, new audio segments are continuously acquired.
[0096] After obtaining new audio clips, clustering is required again to form a new set of speaker clips. The labels of the newly formed set of speaker clips may be inconsistent with the labels of the previously formed set of speaker clips, but may correspond to the same speaker. Therefore, label matching is required, and the previous labels shall prevail.
[0097] For details, see Figure 4 , Figure 4 A flowchart of a tag matching process provided by an embodiment of the present disclosure is shown as follows: Figure 4 As shown, the method also includes:
[0098] S401: Perform voice determination and feature extraction on a second audio segment of the target audio to obtain speaker features of a plurality of subsequent voice segments.
[0099] The second audio segment may be a newly acquired segment of the target audio, and also a segment after the first audio segment.
[0100] First, for the second audio segment of the target audio, human voice determination and feature extraction are also performed. The specific steps can be found in S201-S202 and will not be repeated here.
[0101] S402: re-clustering the speaker segment set, the non-speaker segment set and the subsequent vocal segment to obtain an updated speaker segment set.
[0102] In one embodiment of the present disclosure, step S402 may include the following steps:
[0103] Step 21: re-clustering the vocal segments and subsequent vocal segments included in the non-speaker segment set to obtain a re-clustered speaker segment set and a re-clustered non-speaker segment set.
[0104] First, the vocal segments and subsequent vocal segments included in the non-speaker segment set obtained by the last clustering are re-clustered to obtain a speaker segment set after the re-clustering and a non-speaker segment set after the re-clustering.
[0105] The above re-clustering process is essentially a rough clustering of the vocal segments and subsequent vocal segments included in the current non-speaker segment set, so a larger feature similarity threshold can be set, for example, equivalent to the first similarity threshold in step S203, that is, only speaker segments with very high similarity are clustered into one category. The specific clustering process can be referred to step S203, which will not be repeated here.
[0106] Step 22: Based on the feature similarity between the speaker segment set before re-clustering and the speaker segment set after re-clustering, the speaker segment set before re-clustering and the speaker segment set after re-clustering are merged to obtain at least one merged speaker segment set.
[0107] Since the set of speaker fragments before re-clustering and the set of speaker fragments after re-clustering may correspond to the same speaker, the set of speaker fragments before re-clustering and the set of speaker fragments after re-clustering can be merged based on the feature similarity between the set of speaker fragments before re-clustering and the set of speaker fragments after re-clustering, that is, merged into the same set of speaker fragments.
[0108] As an example, the speaker segment set before re-clustering includes P1 and P2, and the speaker segment set after re-clustering includes P3 and P4, that is, the feature similarities between P1 and P3, P1 and P4, P2 and P3, and P2 and P4 can be calculated respectively, and merged based on the feature similarity. If P1 and P4 actually correspond to the same speaker, the feature similarity between P1 and P4 must be greater than a preset threshold, and P1 and P4 are merged. Since P1 is the speaker segment set before re-clustering, all vocal segments contained in P4 can be divided into P1 to achieve merging.
[0109] In the disclosed embodiment, the similarity between the mean features of the vocal segments respectively included in the two speaker segment sets may be used as the feature similarity between the two speaker segment sets.
[0110] The speaker segment set that is retained after the above merging operation is recorded as the merged speaker segment set. Continuing with the above example, since P4 is merged into P1, the merged speaker segment set includes: P1, P2 and P3.
[0111] Step 23: For each vocal segment contained in the re-clustered non-speaker segment set, calculate the feature similarity between the vocal segment and each merged speaker segment set, and determine the maximum feature similarity therefrom; if the determined maximum feature similarity is greater than a preset third similarity threshold, divide the vocal segment into the merged speaker segment set corresponding to the maximum feature similarity, and obtain an updated speaker segment set.
[0112] In the disclosed embodiment, fine clustering is also performed on each vocal segment included in the re-clustered non-speaker segment set, that is, it is divided into the merged speaker segment set to obtain an updated speaker segment set.
[0113] The basis for division is also feature similarity, and details can be found in step S204, which will not be described in detail.
[0114] At this point, an updated speaker segment set, that is, a new speaker separation result, can be obtained.
[0115] It can be seen that in the embodiment of the present disclosure, segments of online audio can be obtained in stages, and speaker separation can be performed in stages until each segment of the online audio is processed. The processing process has low latency and strong real-time performance, and is thus well suited for real-time speaker separation of online audio.
[0116] S403: Generate a second speaker tag for each updated speaker segment set, and match the speaker segment set before the update with the number of identical vocal segments in the speaker segment set after the update to obtain a matching result of the speaker segment set before the update and the speaker segment set after the update.
[0117] After obtaining the updated speaker segment set, a second speaker tag is generated for each updated speaker segment set.
[0118] As described above, the second speaker label of the updated speaker segment set may be inconsistent with the first speaker label of the previously formed speaker segment set, but may correspond to the same speaker. Therefore, label matching is required, and the previous label, that is, the first speaker label, shall prevail.
[0119] For example, the second speaker label of a certain speaker segment set after update is 2, and the first speaker label of a certain speaker segment set formed previously is 1, but these two speaker segment sets actually correspond to the same speaker. Therefore, it is unreasonable to use different labels, and the label generated for the first time should be used as the basis.
[0120] Specifically, the label matching is based on the number of identical vocal segments in the speaker segment set before the update and the speaker segment set after the update.
[0121] This matching problem can be regarded as a maximum weighted bilateral matching problem and can be solved using the Hungarian algorithm.
[0122] S404: Adjust the second speaker label of the updated speaker segment set to match the first speaker label of the speaker segment set before updating.
[0123] After the matching is completed, the second speaker label of the updated speaker segment set is adjusted to match the first speaker label of the speaker segment set before the update, thereby ensuring the consistency of the generated labels in the process of processing the online audio in stages.
[0124] Combine the following Figure 5 , the speaker separation method provided by the embodiment of the present disclosure is further described, Figure 5 A schematic diagram of a speaker separation method provided by an embodiment of the present disclosure is shown in FIG. Figure 5 As shown, for the target audio (online audio or offline audio) that needs to be separated from the speaker, the target audio is framed, speech detection is performed, and environmental noise and silent frames are removed to obtain vocal segments; then the speaker features of the vocal segments are extracted. On the one hand, based on the speaker features, a bottom-up hierarchical clustering algorithm based on incremental updates is used for coarse clustering to obtain a set of speaker segments and a set of non-speaker segments; on the other hand, an undirected graph of speaker features is constructed based on the speaker features, and then fine clustering is performed based on the undirected graph. Subsequently, label matching is performed based on the previously obtained labels, and the newly generated labels are aligned with the previously generated labels to ensure the consistency of the generated labels in the staged processing of the online audio. Finally, the speaker separation result is obtained.
[0125] Since the processes of feature extraction, coarse clustering and fine clustering can all be performed online, and after coarse clustering and fine clustering, the clustering results can include multiple speaker segment sets, the speaker separation method provided by the embodiment of the present disclosure is suitable for online speaker separation of online audio containing the voices of multiple speakers, and still has a good speaker separation effect when there are many speakers. In addition, there is no need for a large amount of labeled data for speaker separation, and the audio samples for training the feature extraction network only need to be labeled as a whole, which can be obtained from public data sets, and the cost is low.
[0126] See also Figure 6 , Figure 6 is a block diagram of a device for implementing the speaker separation method of the embodiment of the present disclosure, such as Figure 6 As shown, the apparatus may include:
[0127] A determination module 601 is used to perform vocal determination on a first audio segment of the target audio to obtain multiple vocal segments;
[0128] A first feature extraction module 602, configured to extract features from the vocal segments to obtain speaker features of each vocal segment;
[0129] The clustering module 603 is used to cluster the vocal segments based on the feature similarity between the speaker features of each vocal segment and a preset first similarity threshold, so as to obtain at least one speaker segment set and at least one non-speaker segment set; wherein the number of vocal segments included in each speaker segment set reaches a preset number and belongs to the same speaker; and the number of vocal segments included in the non-speaker segment set does not reach the preset number;
[0130] The calculation module 604 is used to calculate the feature similarity between each vocal segment contained in the non-speaker segment set and each speaker segment set, and determine the maximum feature similarity therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, the vocal segment is divided into the speaker segment set corresponding to the maximum feature similarity.
[0131] In one embodiment of the present disclosure, it also includes:
[0132] A construction module, configured to construct an undirected graph based on the feature similarity between the speaker features of each vocal segment, wherein the nodes of the undirected graph are the vocal segments, the connection weights between the nodes are the feature similarities between the vocal segments, and the feature similarities between the connected nodes in the undirected graph are all greater than a preset similarity threshold;
[0133] The calculation module 604 may be specifically used for:
[0134] For each vocal segment included in the non-speaker segment set, based on the number of connections between the vocal segment and the vocal segments in each speaker segment set, a speaker segment set having a number of connections greater than a preset number is screened as a candidate speaker segment set for the vocal segment;
[0135] For each vocal segment included in the non-speaker segment set, the mean similarity between the vocal segment and the vocal segments in each candidate speaker segment set is calculated, which are respectively used as the feature similarities between the vocal segment and each candidate speaker segment set, and the maximum feature similarity is determined from the feature similarities between the vocal segment and each candidate speaker segment set.
[0136] In one embodiment of the present disclosure, it also includes:
[0137] The first generating module is used to generate a first speaker label for each speaker segment set to obtain a speaker separation result for the first audio segment of the target audio.
[0138] In one embodiment of the present disclosure, it also includes:
[0139] A second feature extraction module is used to perform voice determination and feature extraction on a second audio segment of the target audio to obtain speaker features of a plurality of subsequent voice segments;
[0140] a re-clustering module, configured to re-cluster the speaker segment set, the non-speaker segment set and the subsequent vocal segment to obtain an updated speaker segment set;
[0141] A second generating module is used to generate a second speaker label for each updated speaker segment set, and match the speaker segment set before the update with the number of identical vocal segments in the speaker segment set after the update to obtain a matching result of the speaker segment set before the update and the speaker segment set after the update;
[0142] The label adjustment module is used to adjust the second speaker label of the updated speaker segment set to match the first speaker label of the speaker segment set before the update.
[0143] In one embodiment of the present disclosure, the re-clustering module is specifically used to:
[0144] Re-clustering the vocal segments and the subsequent vocal segments included in the non-speaker segment set to obtain a re-clustered speaker segment set and a re-clustered non-speaker segment set;
[0145] Based on the feature similarity between the speaker segment set before re-clustering and the speaker segment set after re-clustering, the speaker segment set before re-clustering and the speaker segment set after re-clustering are merged to obtain at least one merged speaker segment set;
[0146] For each vocal segment contained in the re-clustered non-speaker segment set, the feature similarity between the vocal segment and each merged speaker segment set is calculated, and the maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset third similarity threshold, the vocal segment is divided into the merged speaker segment set corresponding to the maximum feature similarity, to obtain the updated speaker segment set.
[0147] Using the speaker separation device provided by the embodiment of the present disclosure, the first audio segment of the target audio is first subjected to vocal determination to obtain a vocal segment, and then the speech features of the vocal segment are extracted. Subsequently, the first clustering, i.e., coarse clustering, is performed based on the feature similarity between the speaker features of the vocal segments, and a set of clusters reaching a preset number is determined as a speaker segment set. The coarse clustering has a low clustering delay and ensures the high purity of each speaker segment set. Then, based on the feature similarity between the vocal segments contained in the non-speaker segment set and each speaker segment set, the vocal segments in the non-speaker segment set are re-divided, thereby achieving further fine clustering, thereby performing real-time and detailed adjustments to the coarse clustering results to obtain real-time and fast speaker separation results.
[0148] Since the feature extraction and clustering processes can be performed online, and the clustering results can include multiple speaker segment sets, the speaker separation method provided by the embodiment of the present disclosure is suitable for online speaker separation of online audio containing multiple speakers, and has a good speaker separation effect even when there are many speakers. In addition, a large amount of labeled data for speaker separation is not required, and the audio samples for training the feature extraction network only need to be labeled as a whole, which can be obtained from public data sets, and the cost is low.
[0149] According to an embodiment of the present disclosure, the present disclosure also provides an electronic device, a readable storage medium and a computer program product.
[0150] The present disclosure provides an electronic device, comprising:
[0151] at least one processor; and
[0152] a memory communicatively connected to the at least one processor; wherein,
[0153] The memory stores instructions executable by the at least one processor, the instructions being executed by the at least one processor to enable the at least one processor to perform a talker separation method.
[0154] The present disclosure provides a non-transitory computer-readable storage medium storing computer instructions, wherein the computer instructions are used to cause the computer to execute a talker separation method.
[0155] The present disclosure provides a computer program product, including a computer program, which implements a talker separation method when executed by a processor.
[0156] Figure 7A schematic block diagram of an example electronic device 700 that can be used to implement an embodiment of the present disclosure is shown. The electronic device is intended to represent various forms of digital computers, such as laptop computers, desktop computers, workstations, personal digital assistants, servers, blade servers, mainframe computers, and other suitable computers. The electronic device can also represent various forms of mobile devices, such as personal digital processing, cellular phones, smart phones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions are merely examples and are not intended to limit the implementation of the present disclosure described and / or required herein.
[0157] like Figure 7 As shown, the device 700 includes a computing unit 701, which can perform various appropriate actions and processes according to a computer program stored in a read-only memory (ROM) 702 or a computer program loaded from a storage unit 708 into a random access memory (RAM) 703. In the RAM 703, various programs and data required for the operation of the device 700 can also be stored. The computing unit 701, the ROM 702, and the RAM 703 are connected to each other via a bus 704. An input / output (I / O) interface 705 is also connected to the bus 704.
[0158] A number of components in the device 700 are connected to the I / O interface 705, including: an input unit 706, such as a keyboard, a mouse, etc.; an output unit 707, such as various types of displays, speakers, etc.; a storage unit 708, such as a disk, an optical disk, etc.; and a communication unit 709, such as a network card, a modem, a wireless communication transceiver, etc. The communication unit 709 allows the device 700 to exchange information / data with other devices through a computer network such as the Internet and / or various telecommunication networks.
[0159] The computing unit 701 may be a variety of general and / or special processing components with processing and computing capabilities. Some examples of the computing unit 701 include, but are not limited to, a central processing unit (CPU), a graphics processing unit (GPU), various dedicated artificial intelligence (AI) computing chips, various computing units running machine learning model algorithms, digital signal processors (DSPs), and any appropriate processors, controllers, microcontrollers, etc. The computing unit 701 performs the various methods and processes described above, such as the speaker separation method. For example, in some embodiments, the speaker separation method may be implemented as a computer software program, which is tangibly contained in a machine-readable medium, such as a storage unit 708. In some embodiments, part or all of the computer program may be loaded and / or installed on the device 700 via the ROM 702 and / or the communication unit 709. When the computer program is loaded into the RAM 703 and executed by the computing unit 701, one or more steps of the speaker separation method described above may be performed. Alternatively, in other embodiments, the computing unit 701 may be configured to perform the speaker separation method in any other appropriate manner (e.g., by means of firmware).
[0160] Various implementations of the systems and techniques described above herein can be implemented in digital electronic circuit systems, integrated circuit systems, field programmable gate arrays (FPGAs), application specific integrated circuits (ASICs), application specific standard products (ASSPs), systems on chips (SOCs), complex programmable logic devices (CPLDs), computer hardware, firmware, software, and / or combinations thereof. These various implementations can include: being implemented in one or more computer programs that can be executed and / or interpreted on a programmable system including at least one programmable processor, which can be a special purpose or general purpose programmable processor that can receive data and instructions from a storage system, at least one input device, and at least one output device, and transmit data and instructions to the storage system, the at least one input device, and the at least one output device.
[0161] The program code for implementing the method of the present disclosure may be written in any combination of one or more programming languages. These program codes may be provided to a processor or controller of a general-purpose computer, a special-purpose computer, or other programmable data processing device, so that the program code, when executed by the processor or controller, enables the functions / operations specified in the flow chart and / or block diagram to be implemented. The program code may be executed entirely on the machine, partially on the machine, partially on the machine and partially on a remote machine as a stand-alone software package, or entirely on a remote machine or server.
[0162] In the context of the present disclosure, a machine-readable medium may be a tangible medium that may contain or store a program for use by or in conjunction with an instruction execution system, device, or equipment. A machine-readable medium may be a machine-readable signal medium or a machine-readable storage medium. A machine-readable medium may include, but is not limited to, an electronic, magnetic, optical, electromagnetic, infrared, or semiconductor system, device, or equipment, or any suitable combination of the foregoing. A more specific example of a machine-readable storage medium may include an electrical connection based on one or more lines, a portable computer disk, a hard disk, a random access memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), an optical fiber, a portable compact disk read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing.
[0163] To provide interaction with a user, the systems and techniques described herein can be implemented on a computer having: a display device (e.g., a CRT (cathode ray tube) or LCD (liquid crystal display) monitor) for displaying information to the user; and a keyboard and pointing device (e.g., a mouse or trackball) through which the user can provide input to the computer. Other types of devices can also be used to provide interaction with the user; for example, the feedback provided to the user can be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user can be received in any form (including acoustic input, voice input, or tactile input).
[0164] The systems and techniques described herein may be implemented in a computing system that includes back-end components (e.g., as a data server), or a computing system that includes middleware components (e.g., an application server), or a computing system that includes front-end components (e.g., a user computer with a graphical user interface or a web browser through which a user can interact with implementations of the systems and techniques described herein), or a computing system that includes any combination of such back-end components, middleware components, or front-end components. The components of the system may be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: a local area network (LAN), a wide area network (WAN), and the Internet.
[0165] A computer system may include a client and a server. The client and the server are generally remote from each other and usually interact through a communication network. The relationship of client and server is generated by computer programs running on respective computers and having a client-server relationship with each other. The server may be a cloud server, a server of a distributed system, or a server combined with a blockchain.
[0166] It should be understood that the various forms of processes shown above can be used to reorder, add or delete steps. For example, the steps recorded in this disclosure can be executed in parallel, sequentially or in different orders, as long as the desired results of the technical solutions disclosed in this disclosure can be achieved, and this document does not limit this.
[0167] The above specific implementations do not constitute a limitation on the protection scope of the present disclosure. It should be understood by those skilled in the art that various modifications, combinations, sub-combinations and substitutions can be made according to design requirements and other factors. Any modification, equivalent substitution and improvement made within the spirit and principle of the present disclosure shall be included in the protection scope of the present disclosure.
Claims
1. A method for separating speakers, comprising: Performing vocal determination on the first audio segment of the target audio to obtain multiple vocal segments; Extracting features from the vocal segments to obtain speaker features of each vocal segment; Based on the feature similarity between the speaker features of each vocal segment and a preset first similarity threshold, the vocal segments are clustered to obtain at least one speaker segment set and at least one non-speaker segment set; wherein the number of vocal segments included in each speaker segment set reaches a preset number and belongs to the same speaker; the number of vocal segments included in the non-speaker segment set does not reach the preset number; For each vocal segment included in the non-speaker segment set, the feature similarity between the vocal segment and each speaker segment set is calculated, and a maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, the vocal segment is classified into the speaker segment set corresponding to the maximum feature similarity; Performing voice determination and feature extraction on the second audio segment of the target audio to obtain speaker features of a plurality of subsequent voice segments; Re-clustering the speaker segment set, the non-speaker segment set, and the subsequent vocal segment to obtain an updated speaker segment set; Generating a second speaker label for each updated speaker segment set, and matching based on the number of identical vocal segments in the speaker segment set before the update and the speaker segment set after the update, to obtain a matching result of the speaker segment set before the update and the speaker segment set after the update; The second speaker label of the updated speaker segment set is adjusted to match the first speaker label of the speaker segment set before the update.
2. The method according to claim 1, further comprising: constructing an undirected graph based on the feature similarity between the speaker features of each vocal segment, wherein the nodes of the undirected graph are the vocal segments, the connection weights between the nodes are the feature similarities between the vocal segments, and the feature similarities between the connected nodes in the undirected graph are all greater than a preset similarity threshold; The step of calculating the feature similarity between each vocal segment contained in the non-speaker segment set and each speaker segment set, and determining the maximum feature similarity, comprises: For each vocal segment included in the non-speaker segment set, based on the number of connections between the vocal segment and the vocal segments in each speaker segment set, a speaker segment set having a number of connections greater than a preset number is screened as a candidate speaker segment set for the vocal segment; For each vocal segment included in the non-speaker segment set, the mean similarity between the vocal segment and the vocal segments in each candidate speaker segment set is calculated, which are respectively used as the feature similarities between the vocal segment and each candidate speaker segment set, and the maximum feature similarity is determined from the feature similarities between the vocal segment and each candidate speaker segment set.
3. The method according to claim 1, further comprising: A first speaker label is generated for each speaker segment set, and a speaker separation result for the first audio segment of the target audio is obtained.
4. The method according to claim 1, wherein: The step of re-clustering the speaker segment set, the non-speaker segment set and the subsequent vocal segment to obtain an updated speaker segment set includes: Re-clustering the vocal segments and the subsequent vocal segments included in the non-speaker segment set to obtain a re-clustered speaker segment set and a re-clustered non-speaker segment set; Based on the feature similarity between the speaker segment set before re-clustering and the speaker segment set after re-clustering, the speaker segment set before re-clustering and the speaker segment set after re-clustering are merged to obtain at least one merged speaker segment set; For each vocal segment contained in the re-clustered non-speaker segment set, the feature similarity between the vocal segment and each merged speaker segment set is calculated, and the maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset third similarity threshold, the vocal segment is divided into the merged speaker segment set corresponding to the maximum feature similarity, to obtain the updated speaker segment set.
5. A speaker separation device, comprising: A determination module, configured to perform vocal determination on a first audio segment of the target audio to obtain a plurality of vocal segments; A first feature extraction module, used to extract features from the vocal segments to obtain speaker features of each vocal segment; A clustering module, configured to cluster the vocal segments based on the feature similarity between the speaker features of the respective vocal segments and a preset first similarity threshold, to obtain at least one speaker segment set and at least one non-speaker segment set; wherein each speaker segment set includes vocal segments of a preset number and belongs to the same speaker; and the non-speaker segment set includes vocal segments of a preset number or less; a calculation module, configured to calculate, for each vocal segment included in the non-speaker segment set, a feature similarity between the vocal segment and each speaker segment set, and determine a maximum feature similarity therefrom; if the determined maximum feature similarity is greater than a preset second similarity threshold, classify the vocal segment into the speaker segment set corresponding to the maximum feature similarity; A second feature extraction module is used to perform voice determination and feature extraction on a second audio segment of the target audio to obtain speaker features of a plurality of subsequent voice segments; a re-clustering module, configured to re-cluster the speaker segment set, the non-speaker segment set and the subsequent vocal segment to obtain an updated speaker segment set; A second generating module is used to generate a second speaker label for each updated speaker segment set, and match the speaker segment set before the update with the number of identical vocal segments in the speaker segment set after the update to obtain a matching result of the speaker segment set before the update and the speaker segment set after the update; The label adjustment module is used to adjust the second speaker label of the updated speaker segment set to match the first speaker label of the speaker segment set before the update.
6. The apparatus according to claim 5, further comprising: A construction module, configured to construct an undirected graph based on the feature similarity between the speaker features of each vocal segment, wherein the nodes of the undirected graph are the vocal segments, the connection weights between the nodes are the feature similarities between the vocal segments, and the feature similarities between the connected nodes in the undirected graph are all greater than a preset similarity threshold; The computing module is specifically used for: For each vocal segment included in the non-speaker segment set, based on the number of connections between the vocal segment and the vocal segments in each speaker segment set, a speaker segment set having a number of connections greater than a preset number is screened as a candidate speaker segment set for the vocal segment; For each vocal segment included in the non-speaker segment set, the mean similarity between the vocal segment and the vocal segments in each candidate speaker segment set is calculated, which are respectively used as the feature similarities between the vocal segment and each candidate speaker segment set, and the maximum feature similarity is determined from the feature similarities between the vocal segment and each candidate speaker segment set.
7. The apparatus according to claim 5, further comprising: The first generating module is used to generate a first speaker label for each speaker segment set to obtain a speaker separation result for the first audio segment of the target audio.
8. The device according to claim 5, wherein: The re-clustering module is specifically used for: Re-clustering the vocal segments and the subsequent vocal segments included in the non-speaker segment set to obtain a re-clustered speaker segment set and a re-clustered non-speaker segment set; Based on the feature similarity between the speaker segment set before re-clustering and the speaker segment set after re-clustering, the speaker segment set before re-clustering and the speaker segment set after re-clustering are merged to obtain at least one merged speaker segment set; For each vocal segment contained in the re-clustered non-speaker segment set, the feature similarity between the vocal segment and each merged speaker segment set is calculated, and the maximum feature similarity is determined therefrom; if the determined maximum feature similarity is greater than a preset third similarity threshold, the vocal segment is divided into the merged speaker segment set corresponding to the maximum feature similarity, to obtain the updated speaker segment set.
9. An electronic device, comprising: at least one processor; as well as a memory communicatively connected to the at least one processor; wherein, The memory stores instructions that can be executed by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform the method according to any one of claims 1 to 4.
10. A non-transitory computer-readable storage medium storing computer instructions, wherein: The computer instructions are used to cause the computer to execute the method according to any one of claims 1-4.
11. A computer program product, comprising a computer program, which, when executed by a processor, implements the method according to any one of claims 1 to 4.
