Speaker diarization method and system

Abstract

The application provides a speaker diary extraction method and system, comprising: pre-processing audio and video respectively, and extracting initial semantic representations of modality-aligned audio data and each speaker video data; obtaining rough division of a speaker active segment by using semantic information of the video, and extracting acoustic features of the speaker from the rough division; modeling uncertainty of each modality at a frame level according to the semantic information and the speaker features, and evaluating quality of the audio and the video; guiding fusion of audio and video modality semantic representations by using a confidence score, and obtaining fused semantic information streams of each speaker audio and video; and estimating active states of each speaker at each time by a decoder according to the fused semantic information streams of each speaker audio and video. The application uses the confidence score to guide dynamic fusion of different modality information, has better robustness to noisy segments compared to other methods, and has good compatibility for multi-channel audio.

Description

Technical Field

[0001] This invention relates to the field of audio and video recognition and analysis technology, specifically to a speaker log extraction method and system based on confidence score and multimodal information flow fusion, and more particularly to a speaker log extraction method and system. Background Technology

[0002] Speaker logging is a fundamental task in intelligent speech processing, aiming to identify "who speaks and when" in an audio clip. By incorporating video modality, the system can acquire richer information about the speaker's identity and activity level. While previous multimodal fusion methods have performed well in specific scenarios, these methods primarily rely on high-quality data and do not consider real-world conditions such as acoustic noise or missing faces. Therefore, designing a new method that maintains robustness to complex real-world environments is an important and highly challenging task.

[0003] Chinese invention patent CN116312552A discloses an audio / video speaker log method and system. The invention's model method separates an audio / video recording into audio and video components. Firstly, it uses speech recognition technology to segment the entire audio portion, obtaining speech segments containing only a single speaker. For each speech segment, speaker attribute detection is performed to obtain speaker attribute information and extract voiceprint features. Secondly, it performs target detection and tracking on the video portion to obtain target person attribute information. Finally, it combines the matching structure of speaker attribute information and target person attribute information with voiceprint feature comparison to determine the speaker.

[0004] This patent does not take into account the speaker overlap problem that exists in real-world scenarios, as well as the poor or even missing audio and video quality caused by equipment, environment and other reasons. Therefore, it cannot effectively cope with the negative impact of complex environments. Summary of the Invention

[0005] To address the shortcomings of existing technologies, this invention provides a method and system for extracting speaker logs.

[0006] According to the present invention, a speaker log extraction method and system are provided, the solution of which is as follows:

[0007] Firstly, a method for extracting speaker logs is provided, the method comprising:

[0008] Step S1: Preprocess the audio and video separately, and extract the initial semantic representations of the modally aligned audio data and the video data of each speaker respectively;

[0009] Step S2: Use the semantic information of the video to obtain a rough division of the speaker's active segments, and extract the speaker's acoustic features accordingly;

[0010] Step S3: Perform frame-level uncertainty modeling for each modality based on semantic information and speaker characteristics, and evaluate the quality of audio and video;

[0011] Step S4: Use confidence scores to guide the fusion of audio and video modal semantic representations to obtain the fused audio and video semantic information streams of each speaker;

[0012] Step S5: The decoder estimates the active state of each speaker at each time step based on the fused audio and video semantic information streams of each speaker.

[0013] Preferably, step S1 includes:

[0014] The audio and video are processed separately by segmentation and frame capture.

[0015] Facial images of each speaker are extracted using a facial recognition algorithm;

[0016] Semantic representations of audio data and speaker face video data are obtained by using modally aligned pre-trained feature extractors for audio and vision, respectively.

[0017] Preferably, step S2 includes:

[0018] The video semantic representation is input into the video decoder to obtain the speaker activity state Y based on the video. V ;

[0019] According to Y V Active segments are extracted from the original audio, and acoustic feature extractors are used to obtain the acoustic representations of each speaker.

[0020] Among them, D I This represents the feature dimension of each representation vector, and N is the total number of speakers appearing in the original audio.

[0021] Preferably, step S3 includes:

[0022] The semantic representation of the audio is concatenated with the acoustic features of each speaker to obtain a speaker-specific audio semantic representation.

[0023] For semantic representation of audio and video data, each frame is modeled as an independent multidimensional Gaussian distribution, denoted as... Its mean represents the modality characteristics, while its variance represents its confidence level;

[0024] Where M∈A,V, they correspond to audio and video modalities, respectively; and These are the dimensions and the original dimension D of the semantic representation of each modality. M The same mean and variance vector; t∈1,2,…,T, where T is the total length of the input data stream;

[0025] Based on the semantic representation, the mean and variance are calculated using two independent feedforward neural networks respectively;

[0026] In calculating the variance, for the output of the feedforward neural network, an exponential function of the natural logarithm e is used to ensure that the variance is always greater than 0.

[0027] Preferably, step S4 includes:

[0028] The audio and video modalities are sampled from a multidimensional Gaussian distribution to obtain their hidden states.

[0029] The missing faces of each speaker in the video modality are recorded as follows: A value of 0 indicates that the speaker's face was not detected in the frame, while a value of 1 indicates the opposite.

[0030] Calculate the confidence score based on the variance of each distribution, expressed as follows:

[0031] Based on the confidence score and the hidden state, the fused semantic representation is obtained:

[0032]

[0033] in, This represents the vector concatenation operation, resulting in... This is the final result of modality fusion in the network.

[0034] Preferably, during training, the distribution is sampled using a reparameterization method: Where ∈~N(0,I), is obtained by sampling the standard Gaussian distribution. This method ensures that the gradient is backpropagated during training.

[0035] During the reasoning process, the mean vector is used directly. As the optimal hidden state vector.

[0036] Preferably, only single-channel audio data is needed during training;

[0037] During inference, for multi-channel audio, each channel is treated as a sub-modality of the audio, and its hidden state vector and variance are calculated separately.

[0038] Calculate the confidence scores between channels, and pre-fuse the audio data and variance of each channel using a weighted average to obtain... and

[0039] Preferably, step S5 includes:

[0040] The fused semantic representations of each speaker are passed through a long short-term recurrent neural network to obtain its intermediate output;

[0041] The intermediate outputs are aligned temporally and concatenated along the feature dimension according to speaker order. This concatenation is then passed through a second long short-term recurrent neural network to obtain a frame-level semantic representation, represented as follows:

[0042] By applying different linear layers and sigmoid operations to the frame-level semantic representation, the activity probability of the nth speaker at time t is obtained:

[0043]

[0044] Among them, W n and b n These are the parameters for the nth linear layer, corresponding to the nth speaker;

[0045] Set a threshold s, when When the speaker is active, it means that the speaker is currently in an active state.

[0046] Preferably, the optimization objectives of the speaker log include:

[0047] Set optimization objective one: fit the predicted results to the true labels.

[0048]

[0049] Among them, M + ∈{A,V,AV} represents the set of single modes before fusion and the total set after fusion, y n,t ∈{0,1} represents the active state of the nth speaker at time t in the actual annotation;

[0050] During training, an additional single-modal decoder is introduced to predict the activity probability based on the semantic representations of each modality before fusion. Compare them with the real label y respectively n,t The task loss is calculated using the cross-entropy loss function.

[0051] The second optimization objective is to ensure that the variance accurately represents the confidence level of the mode:

[0052] First, the cross-entropy function is used to calculate the distance between the single-modal predicted value and the true value.

[0053] Perform L2 regularization on the variance vector and calculate its reciprocal:

[0054] The distance sequences and variance vector sequences of different modalities are concatenated along the time dimension, i.e.:

[0055] Will and Normalization is performed using the Softmax operation on the time dimension, and the KL divergence is calculated to obtain the confidence loss. Represented as:

[0056]

[0057] The third optimization objective is to introduce a regularization term to prevent training crashes.

[0058]

[0059] Where μ and σ are the mean and variance of each multidimensional Gaussian distribution, respectively, and I is the identity matrix.

[0060] The final total loss function is:

[0061]

[0062] Where, λ c With λ r The loss weight hyperparameter is set manually.

[0063] Secondly, a speaker log extraction system is provided, the system comprising:

[0064] Audio and video feature extraction module: The audio and video are segmented and frame-captured separately, and face images are extracted using a face recognition algorithm; the semantic representations of aligned audio and video data are obtained using a pre-trained audio feature extractor and visual feature extractor.

[0065] Speaker acoustic feature extraction module: Inputs the video semantic representation into the video decoder to obtain a rough division of the speaker's active segments, and extracts the speaker's acoustic features from them;

[0066] Uncertainty modeling module: Based on semantic information and acoustic features, audio and video modalities are modeled as speaker-specific independent Gaussian distributions frame by frame;

[0067] Modality fusion module: Calculates modality confidence scores based on the variance of the distribution and the modality missing information, and uses the confidence scores to guide the fusion of audio and video modality semantic representations to obtain the fused audio and video information streams of each speaker;

[0068] Active state prediction module: Predicts the active probability of each speaker at each time step based on the fused representation.

[0069] Compared with the prior art, the present invention has the following beneficial effects:

[0070] 1. This invention effectively improves the accuracy of multimodal speaker log systems in real-world scenarios;

[0071] 2. This invention utilizes uncertainty to guide the fusion of audio and video modalities. Compared with other fusion methods, it can effectively handle low-quality segments of each modality in the data, thus being more robust to noisy and incomplete far-field data.

[0072] 3. This invention can utilize the internal modal uncertainty learned from single-channel data to achieve good performance on multi-channel data without additional training.

[0073] Other beneficial effects of the present invention will be explained in detail through the introduction of specific technical features and technical solutions in specific embodiments. Those skilled in the art should be able to understand the beneficial technical effects brought about by these technical features and technical solutions through the introduction of these technical features and technical solutions. Attached Figure Description

[0074] Other features, objects, and advantages of the present invention will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings:

[0075] Figure 1 This is a schematic diagram of the method flow of the present invention;

[0076] Figure 2 This is a schematic diagram of the system of the present invention.

[0077] Specific implementation party

[0078] The present invention will now be described in detail with reference to specific embodiments. These embodiments will help those skilled in the art to further understand the present invention, but do not limit the invention in any way. It should be noted that those skilled in the art can make several changes and improvements without departing from the concept of the present invention. These all fall within the protection scope of the present invention.

[0079] This invention provides a method for extracting speaker logs, which specifically includes the following:

[0080] Semantic representations of modally aligned audio data and video data of each speaker are extracted separately.

[0081] By utilizing the semantic information of the video, a rough segmentation of the speaker's active segments is obtained, and the speaker's acoustic features are extracted from this segment.

[0082] Frame-level uncertainty modeling is performed on each modality based on semantic information and speaker characteristics, and the quality of audio and video is dynamically evaluated.

[0083] The confidence scores are used to guide the fusion of audio and video modal semantic representations, resulting in a fused audio-visual semantic information stream for each speaker.

[0084] The decoder estimates the active state of each speaker at each moment based on audio and video information. This invention, by utilizing confidence-guided modality fusion, can reduce the negative impact of low-quality segments on prediction results and improve accuracy in real-world scenarios.

[0085] Further optimizations based on the above embodiments, such as... Figures 1-2 The diagram shown is a flowchart of the method in this embodiment, including:

[0086] Step S1: Preprocess the audio and video separately, and extract the semantic representations of the modally aligned audio data and the video data of each speaker.

[0087] Specifically, in step S1:

[0088] The audio and video are processed separately by segmentation and frame capture.

[0089] Facial images of each speaker are extracted using a facial recognition algorithm;

[0090] Semantic representations of audio data and speaker facial data are obtained by using modally aligned pre-trained feature extractors for audio and vision, respectively.

[0091] Step S2: Use the semantic information of the video to obtain a rough division of the speaker's active segments, and extract the speaker's acoustic features from this.

[0092] Specifically, in step S2:

[0093] The video semantic representation is input into the video decoder to obtain the speaker activity state Y based on the video. V ;

[0094] According to Y V Active segments are extracted from the original audio, and acoustic feature extractors are used to obtain the acoustic representations of each speaker. Where D IThe feature dimension of each representation vector is represented by N, which is the total number of speakers appearing in the original audio.

[0095] Step S3: Based on semantic information and speaker characteristics, perform frame-level uncertainty modeling for each modality, and dynamically evaluate the quality of audio and video.

[0096] Specifically, in step S3:

[0097] The semantic representation of the audio is concatenated with the acoustic features of each speaker to obtain a speaker-specific audio semantic representation.

[0098] For semantic representation of audio and video data, each frame is modeled as an independent multidimensional Gaussian distribution, denoted as... Its mean represents the modality characteristics, while its variance represents its confidence level; where M∈A,V, corresponding to audio and video modalities, respectively; and These are the dimensions and the original dimension D of the semantic representation of each modality. M The same mean and variance vector; t∈1,2,…,T, where T is the total length of the input data stream.

[0099] Specifically, the modeling of the multidimensional Gaussian distribution includes:

[0100] Based on the semantic representation, the mean and variance are calculated using two independent feedforward neural networks respectively;

[0101] In calculating the variance, for the output of the feedforward neural network, an exponential function of the natural logarithm e is used to ensure that the variance is always greater than 0.

[0102] Step S4: Use confidence scores to guide the fusion of audio and video modal semantic representations to obtain the fused audio and video semantic information streams of each speaker.

[0103] Specifically, in step S4:

[0104] The audio and video modalities are sampled from a multidimensional Gaussian distribution to obtain their hidden states.

[0105] The missing faces of each speaker in the video modality are recorded as follows: A value of 0 indicates that the speaker's face was not detected in the frame, while a value of 1 indicates the opposite.

[0106] Calculate the confidence score based on the variance of each distribution, expressed as follows:

[0107] Based on the confidence score and the hidden state, the fused semantic representation is obtained:

[0108]

[0109] in, This represents the vector concatenation operation. The result is... This is the final result of modality fusion in the network.

[0110] Specifically, the sampling process for a multidimensional Gaussian distribution includes:

[0111] During training, the distribution is sampled using a reparameterization method: Where ∈~N(0,I), is obtained by sampling the standard Gaussian distribution. This method ensures that the gradient can propagate backward during training.

[0112] During the reasoning process, the mean vector is used directly. As the optimal hidden state vector.

[0113] Specifically, the processing of multi-channel audio includes:

[0114] During training, only single-channel audio data is required;

[0115] During inference, for multi-channel audio, each channel is treated as a sub-modality of the audio, and its hidden state vector and variance are calculated separately.

[0116] Calculate the confidence scores between channels, and pre-fuse the audio data and variance of each channel using a weighted average method to obtain the final result. and

[0117] Step S5: The decoder estimates the active state of each speaker at each moment based on the audio and video information.

[0118] Specifically, in step S5:

[0119] The fused semantic representations of each speaker are passed through a long short-term recurrent neural network to obtain its intermediate output;

[0120] The intermediate outputs are aligned temporally and concatenated along the feature dimension according to speaker order. This concatenation is then passed through a second long short-term recurrent neural network to obtain a frame-level semantic representation, represented as follows:

[0121] By applying different linear layers and sigmoid operations to the frame-level semantic representation, the activity probability of the nth speaker at time t is obtained:

[0122]

[0123] Among them, W n and b nThese are the parameters for the nth linear layer, corresponding to the nth speaker;

[0124] Set a threshold s, when When the speaker is active, it means that the speaker is currently in an active state.

[0125] Specifically, the optimization goals for speaker logs include:

[0126] Set optimization objective one: fit the predicted results to the true labels.

[0127]

[0128] Among them, M + ∈{A,V,AV} represents the set of single modes before fusion and the total set after fusion, y n,t ∈{0,1} represents the active state of the nth speaker in the ground truth annotation at time t. During training, a single-modal decoder is additionally introduced to predict the active probability based on the semantic representations of each modality before fusion. Compare them with the real label y respectively n,t The task loss is calculated using the cross-entropy loss function.

[0129] The second optimization objective is to ensure that the variance accurately represents the confidence level of the mode:

[0130] First, the cross-entropy function is used to calculate the distance between the single-modal predicted value and the true value.

[0131] Perform L2 regularization on the variance vector and calculate its reciprocal:

[0132] The distance sequences and variance vector sequences of different modalities are concatenated along the time dimension, i.e.:

[0133] Will and Normalization is performed using the Softmax operation on the time dimension, and the KL divergence is calculated to obtain the confidence loss. Represented as:

[0134]

[0135] The third optimization objective is to introduce a regularization term to prevent training crashes.

[0136]

[0137] Where μ and σ are the mean and variance of each multidimensional Gaussian distribution, respectively, and I is the identity matrix.

[0138] The final total loss function is:

[0139]

[0140] Where, λ c With λ r The loss weight hyperparameter is set manually.

[0141] This invention also provides a speaker log extraction system, such as Figure 2 The diagram shown is a schematic of the system in this embodiment, including:

[0142] Audio and video feature extraction module: The audio and video are segmented and frame-captured separately, and face images are extracted using a face recognition algorithm; the semantic representations of aligned audio and video data are obtained using a pre-trained audio feature extractor and visual feature extractor.

[0143] Speaker acoustic feature extraction module: Inputs the video semantic representation into the video decoder to obtain a rough division of the speaker's active segments, and extracts the speaker's acoustic features from them;

[0144] Uncertainty modeling module: Based on semantic information and acoustic features, audio and video modalities are modeled as speaker-specific independent Gaussian distributions frame by frame;

[0145] Modality fusion module: Calculates modality confidence scores based on the variance of the distribution and the modality missing information, and uses the confidence scores to guide the fusion of audio and video modality semantic representations to obtain the fused audio and video information streams of each speaker;

[0146] Active state prediction module: Predicts the active probability of each speaker at each time step based on the fused representation.

[0147] This invention can utilize publicly available datasets to generate dialogues in multimodal scenarios, achieving excellent text generation metrics.

[0148] This invention provides a speaker log extraction method and system that uses confidence scores to guide the dynamic fusion of different modal information. Compared with other methods, it has better robustness to noisy segments and good compatibility with multi-channel audio.

[0149] Those skilled in the art will understand that, besides implementing the system and its various devices, modules, and units provided by this invention in the form of purely computer-readable program code, the same functions can be achieved entirely through logical programming of the method steps, making the system and its various devices, modules, and units of this invention function in the form of logic gates, switches, application-specific integrated circuits, programmable logic controllers, and embedded microcontrollers. Therefore, the system and its various devices, modules, and units provided by this invention can be considered as a hardware component, and the devices, modules, and units included therein for implementing various functions can also be considered as structures within the hardware component; alternatively, the devices, modules, and units for implementing various functions can be considered as both software modules implementing the method and structures within the hardware component.

[0150] Specific embodiments of the present invention have been described above. It should be understood that the present invention is not limited to the specific embodiments described above, and those skilled in the art can make various changes or modifications within the scope of the claims, which do not affect the essence of the present invention. Unless otherwise specified, the embodiments and features described in this application can be arbitrarily combined with each other.

Claims

1. A method for extracting speaker logs, characterized in that, include: Step S1: Preprocess the audio and video separately, and extract the semantic representations of the modally aligned audio data and the video data of each speaker. Step S2: Use the semantic information of the video to obtain a rough division of the speaker's active segments, and extract the speaker's acoustic features accordingly; Step S3: Perform frame-level uncertainty modeling for each modality based on semantic information and speaker acoustic features, and evaluate the quality of audio and video; Step S4: Use confidence scores to guide the fusion of audio and video modal semantic representations to obtain the fused audio and video semantic information streams of each speaker; Step S5: The decoder estimates the active state of each speaker at each moment based on the fused audio and video semantic information streams of each speaker; Step S3 includes: The semantic representation of the audio is concatenated with the acoustic features of each speaker to obtain a speaker-specific audio semantic representation. For the semantic representation of audio and video data, each frame is modeled as an independent multidimensional Gaussian distribution, denoted as . The mean represents the modality characteristics, while the variance represents its confidence level. in, These correspond to audio and video modalities, respectively. and These are the dimensions and the original dimensions of the semantic representation of each modality. The same mean and variance vector; ,in The total length of the input data stream; Based on the semantic representation, the mean and variance are calculated using two independent feedforward neural networks respectively; In calculating the variance, the natural logarithm is additionally used for the output of the feedforward neural network. An exponential function is used to ensure that the variance is always greater than 0; Step S4 includes: The audio and video modalities are sampled from a multidimensional Gaussian distribution to obtain their hidden states. , ; The missing faces of each speaker in the video modality are recorded as follows: A value of 0 indicates that the speaker's face was not detected in the frame, while a value of 1 indicates the opposite. Calculate the confidence score based on the variance of each distribution, expressed as follows: , ; Based on the confidence score and the hidden state, the fused semantic representation is obtained: in, This represents the vector concatenation operation, resulting in... This is the final result of modality fusion in the network; Step S5 includes: The fused semantic representations of each speaker are passed through a long short-term recurrent neural network to obtain its intermediate output; The intermediate outputs are aligned temporally and concatenated along the feature dimension according to speaker order. This concatenation is then passed through a second long short-term recurrent neural network to obtain a frame-level semantic representation, represented as follows: ; Applying different linear layers and Sigmoid operations to the frame-level semantic representation yields the... The probability of a speaker being active at time t: in, and For the first The parameters of the nth linear layer, that is, the parameters corresponding to the nth linear layer. One speaker; Set threshold ,when When the speaker is active, it means that the speaker is currently in an active state.

2. The speaker log extraction method according to claim 1, characterized in that, Step S1 includes: The audio and video are processed separately by segmentation and frame capture. Facial images of each speaker are extracted using a facial recognition algorithm; Semantic representations of audio data and speaker face video data are obtained by using modally aligned pre-trained feature extractors for audio and vision, respectively.

3. The speaker log extraction method according to claim 1, characterized in that, Step S2 includes: The video semantic representation is input into the video decoder to obtain the speaker's active state based on the video. ; according to Active segments are extracted from the original audio, and acoustic feature extractors are used to obtain the acoustic representations of each speaker. ; in, This represents the feature dimension of each representation vector. This represents the total number of speakers appearing in the original audio.

4. The speaker log extraction method according to claim 1, characterized in that, During training, the distribution is sampled using a reparameterization method: ,in The gradient is obtained by sampling a standard Gaussian distribution, and this method ensures that the gradient is backpropagated during training. During the reasoning process, the mean vector is used directly. As the optimal hidden state vector.

5. The speaker log extraction method according to claim 1, characterized in that, During training, only single-channel audio data is required; During inference, for multi-channel audio, each channel is treated as a sub-modality of the audio, and its hidden state vector and variance are calculated separately. Calculate the confidence scores between channels, and pre-fuse the audio data and variance of each channel using a weighted average to obtain... and .

6. A speaker log extraction system, based on the speaker log extraction method according to any one of claims 1-5, characterized in that, include: Audio and video feature extraction module: Performs segmentation and frame acquisition processing on audio and video respectively, and extracts face images using face recognition algorithms; Semantic representations of aligned audio and video data are obtained using pre-trained audio and visual feature extractors. Speaker acoustic feature extraction module: Inputs the video semantic representation into the video decoder to obtain a rough division of the speaker's active segments, and extracts the speaker's acoustic features from them; Uncertainty modeling module: Based on semantic information and acoustic features, audio and video modalities are modeled as speaker-specific independent Gaussian distributions frame by frame; Modality fusion module: Calculates modality confidence scores based on the variance of the distribution and the modality missing information, and uses the confidence scores to guide the fusion of audio and video modality semantic representations to obtain the fused audio and video information streams of each speaker; Active state prediction module: Predicts the active probability of each speaker at each time step based on the fused representation.

Images