Method and system for actively predicting next speaker in multi-person dialogue scene

Abstract

The invention discloses an active prediction method and system for a next speaker in a multi-person dialogue scene, and belongs to the field of natural language processing and robot dialogue decision making. The method aims at solving the problems that an existing robot is passive in interaction, weak in new speaker recognition and low in model efficiency. The core scheme comprises the steps of enhancing speaker dynamic representation generation based on multi-source gain fusion, fusing three types of gains of long-term behaviors, dynamic time sequences and global semantics, and constructing context-aware participant portraits. A single-stage unified modeling architecture is adopted, historical speaker classification and new speaker detection are jointly optimized, and error propagation is avoided; and a self-adaptive asymmetric focus loss function relieves class imbalance caused by sparse new speaker samples. According to the method, high calculation efficiency and new speaker recognition capability in an open set scene are both considered, the robot is upgraded from passive response to active participants, and the method is suitable for group chat assistants, conference assistants and the like.

Description

Technical Field

[0001] This invention belongs to the field of intelligent robot technology, specifically relating to a method and system for actively predicting the next speaker in a multi-person dialogue scenario. This method aims to upgrade the robot from a "passive response" mode to an "active participant" by accurately predicting the direction of the dialogue and autonomously planning the timing and target of its speech. Background Technology

[0002] With the increasing prevalence of service robots, social robots, and intelligent conferencing systems, robots are participating more and more in multi-person natural conversations. To achieve human-like, highly interactive intelligent agents, robots not only need to understand the content of the conversation but also need to master the social rules and rhythm of human dialogue, knowing when to speak and to whom. However, existing robot systems still face many core bottlenecks in these scenarios, seriously affecting the interactive experience and practical value.

[0003] Current robots generally adopt a "trigger-response" interaction mode, only responding when directly @mentioned, called by name, or when specific keywords are detected. This passive mode leads to rigid robot behavior, making it unable to proactively engage in conversation, guide the topic, or achieve smooth transitions in dialogue like human participants. The interaction experience is stiff, lacks human-like qualities, and is difficult to truly integrate into real conversational processes.

[0004] Lacking the ability to predict the next speaker, robots cannot plan optimal interaction strategies. In practical applications, two common dilemmas arise: first, it's difficult to accurately predict the window of opportunity for dialogue breaks, leading to missed opportunities to speak; second, it may mistakenly trigger a response before someone has finished speaking, interrupting the conversation, violating basic social etiquette, and impacting communication efficiency and experience.

[0005] The robot system needs to continuously receive and process audio / text stream data from all participants, but only outputs responses in a few triggered scenarios, resulting in a significant waste of computing resources. Especially in embedded or mobile robot platforms, this inefficient operating mode exacerbates power consumption pressure and makes it difficult to meet the performance requirements of real-time interaction, limiting the deployment and application of robots in more scenarios.

[0006] Some current research focuses on "Next Speaker Prediction (NSP)," but when these academic solutions are directly applied to robotic systems, they still have key technical deficiencies and cannot meet the needs of engineering applications.

[0007] Existing solutions largely rely on general text encoders to extract features, failing to effectively integrate core information about dialogue participants, including long-term behavioral roles (such as conversation leaders and followers), dynamic temporal activity levels (such as silence duration and speaking intervals), and deep semantic connections in the dialogue. This results in the robot obtaining an overly simplistic speaker profile, which cannot support refined interactive decision-making, and the accuracy of predictions is difficult to guarantee.

[0008] In real-world conversations, new participants frequently join the conversation, but samples of these new participants are extremely scarce in the training data. Standard training objectives can cause the model to develop a strong preference for past speakers, making it difficult for the robot to recognize new participants in a timely manner. This results in poor adaptability in dynamic and open scenarios such as meetings and group chats, severely limiting the robot's application scope.

[0009] Most existing solutions employ a two-stage cascaded architecture: first detecting whether the speaker is a previous speaker, then identifying the specific identity. This architecture not only incurs high computational overhead, but errors in the front-end detection module also propagate directly to the back-end recognition stage, resulting in high overall prediction latency and reduced reliability. It fails to meet the stringent requirements of low-latency response for real-time robot interaction and is difficult to deploy on resource-constrained robot platforms.

[0010] Therefore, there is an urgent need in this field for a lightweight, efficient, and robot-optimized next speaker prediction solution. This solution should provide the robot with context-aware, dynamic participant profiles and fundamentally solve the challenge of new speaker identification, thereby enabling the robot to achieve fluent, proactive, and human-like interaction in open, dynamic multi-person dialogue environments. Summary of the Invention

[0011] The purpose of this invention is to overcome the shortcomings of existing next speaker prediction (NSP) technology and provide a method and system for proactive next speaker prediction specifically designed for multi-person dialogue robot systems. By constructing context-aware enhanced speaker representations, adopting an efficient single-stage prediction architecture, and utilizing an optimization strategy oriented towards class imbalance, it achieves high-precision, low-latency, and highly robust proactive prediction, especially improving the ability to accurately identify new speakers, thereby providing core decision support for the robot's speech planning, attention allocation, and resource scheduling.

[0012] To achieve the above objectives, the present invention adopts the following technical solution:

[0013] In a first aspect, the present invention provides a method for actively predicting the next speaker in a multi-person dialogue scenario, comprising the following steps:

[0014] S1. Load formatted multi-person dialogue data, construct context windows and training samples, generate a set of candidate speakers and calculate the dynamic temporal features and global long-term behavioral features of each candidate.

[0015] S2. A pre-trained language model based on Transformer is used as a context encoder. Multi-source gain fusion is used to generate enhanced speaker dynamic representations. The matching scores between the global semantic vector and each candidate representation are calculated and the prediction results are output.

[0016] S3. An adaptive asymmetric focus loss function is used to alleviate the class imbalance problem, and the model training is completed by combining differentiated learning rate scheduling, oversampling and early stopping strategies.

[0017] As a preferred technical solution, step S1 specifically includes:

[0018] S101: Read multi-person dialogue data from a JSONL format file, where each dialogue contains multiple rounds of statements with speaker identifiers;

[0019] S102: Define the size of the context window, extract the corresponding round range as the current context window, maintain the complete speaker's speaking position sequence, splice the sentences in the window to form the model input text, and use the next round speaker as the prediction target;

[0020] S103: Construct a candidate speaker set, which includes all unique historical speakers up to the current round and generalized identifiers representing new speakers; calculate the relative positions of each candidate within the window as dynamic temporal features, and calculate multidimensional global long-term behavioral features based on the complete speaker speaking position sequence.

[0021] As a preferred technical solution, the multidimensional global long-term behavioral characteristics include: global speaking frequency, normalized nearest speaking distance, local first speaker, candidate status identifier, normalized average turn length, speaking dominance, recent responsiveness, and speaking consistency.

[0022] Wherein, the global speaking frequency is the ratio of the total number of times a speaker appears in the complete speaker speaking position sequence to the total number of historical rounds;

[0023] The normalized nearest speaking distance is the normalized result of the difference between the current round and the speaker's last speaking round relative to the length of the complete speaker speaking position sequence;

[0024] The local first speaker is a binary feature that indicates whether the speaker is the first speaker in the current context window;

[0025] The candidate status identifier is a discrete numerical feature used to indicate the candidate's existence status in the historical record and the current window, including three values: active historical state, regressive latent state, and unknown new ecosystem.

[0026] The normalized average turn length is the normalized result of the average number of words in all of the speaker's historical speech relative to a preset normalization factor;

[0027] The speaking dominance is the proportion of a speaker’s total historical speaking content to the total content of all speakers, and the content is calculated based on the number of words or phrases.

[0028] The recent responsiveness is a binary feature used to determine whether the speaker spoke in the previous round immediately preceding the current round.

[0029] The consistency of speech is calculated based on the standard deviation of the speaker's historical speech length sequence and is used to reflect the stability of the speaker's speech length.

[0030] As a preferred technical solution, the multi-source gain fusion in step S2 includes deep fusion of long-term behavioral gain, dynamic temporal gain, and global semantic gain. The specific process is as follows:

[0031] S21: Input the input text into the pre-trained language model encoder and extract the hidden state corresponding to the [CLS] tag as the global semantic gain;

[0032] S22: Input multidimensional long-term behavioral features into a two-layer MLP and output dynamic role gain; map relative position values ​​through a learnable position embedding matrix and output dynamic temporal gain;

[0033] S23: The basic gain feature is obtained by splicing dynamic role gain and dynamic temporal gain and passing it through a linear fusion layer. The attention enhancement feature is output through a multi-head attention module. The attention enhancement feature and the basic gain feature are dynamically fused through a gating network to obtain the enhanced basic feature.

[0034] S24: Concatenate global semantic gain, enhanced basic features and attention-enhanced features, and output the final enhanced speaker dynamic representation through a context-aware fusion network.

[0035] As a preferred technical solution, the prediction result in step S2 is obtained in the following way:

[0036] The dot product similarity between the global semantic vector and each candidate representation is calculated as the matching score. The Softmax function is applied to all candidate scores to obtain the probability distribution, and the candidate with the highest probability is selected as the prediction result.

[0037] As a preferred technical solution, the adaptive asymmetric focus loss function in step S3 is constructed based on the label-smoothed basic cross-entropy loss, and introduces a dynamic weighting factor and a dynamic focus index. Both the dynamic weighting factor and the dynamic focus index are linear interpolation functions of the true class prediction probability. By setting asymmetric parameter relationships, attention to low-confidence samples is enhanced.

[0038] As a preferred technical solution, the final loss function for:

[0039]

[0040] in, The predicted probability of the true class. As a weighting factor, To focus on the index, The basic cross-entropy loss is labeled and smoothed. The calculation is as follows:

[0041]

[0042] in, The label smoothing coefficient. Indicates a candidate speaker. This represents the set of candidate speakers.

[0043] As a preferred technical solution, the differentiated learning rate scheduling in step S3 specifically involves: using a lower learning rate for the pre-trained language model encoder parameters, using a higher learning rate for the newly added module parameters, and employing a linear learning rate decay strategy with preheating.

[0044] Secondly, the present invention provides a system for proactively predicting the next speaker in a multi-person dialogue scenario, used to implement the aforementioned method for proactively predicting the next speaker in a multi-person dialogue scenario, characterized in that it includes:

[0045] An audio acquisition unit is used to acquire real-time audio signals in a dialogue scenario using a microphone array.

[0046] The speech recognition unit is used to receive the audio signal output by the audio acquisition unit, convert it into a real-time text stream and transmit it to the electronic device for active prediction of the next speaker;

[0047] The speech judgment unit is used to receive the probability distribution of the predicted output and determine whether the next speaker is a robot character;

[0048] The natural language generation unit is used to generate a text response when the next speaker is determined to be a robot.

[0049] A speech synthesis unit is used to convert the text response into a speech signal; a speech playback unit is used to receive the speech signal and play it through a speaker.

[0050] Thirdly, the present invention provides an electronic device, the electronic device comprising:

[0051] At least one processor; and,

[0052] A memory communicatively connected to the at least one processor; wherein,

[0053] The memory stores computer program instructions that can be executed by the at least one processor, which enables the at least one processor to perform the next speaker active prediction method in the multi-person dialogue scenario.

[0054] Compared with the prior art, the present invention has the following advantages and beneficial effects:

[0055] (1) This invention upgrades robots from "passive response" to "active prediction and planning". The robot can predict the direction of the conversation, thereby proactively planning the best time to speak, selecting the responder, and even preparing relevant content resources in advance. This is the core technology for building the next generation of robots with social intelligence and proactive interaction capabilities.

[0056] (2) This invention generates speaker representations based on dynamic role modeling of long-term behavioral memory, enabling the robot to deeply understand the role and state of each participant in the dialogue. The model completely eliminates the dependence on fixed speaker IDs, significantly improving the prediction accuracy and adaptability of the robot in open sets and real-world scenarios without prior identity knowledge.

[0057] (3) The single-stage unified modeling framework of this invention is computationally efficient, fundamentally avoiding the latency and error accumulation of traditional cascaded architectures. Combined with a lightweight model design, it ensures that predictions can be completed within milliseconds, fully meeting the requirements of real-time chatbots for low-latency response, and facilitating deployment on resource-constrained embedded platforms.

[0058] (4) This invention systematically solves the problem of new speaker recognition, which has plagued practical robot applications, by using an adaptive asymmetric focus loss function. It significantly improves the robot's sensitivity to new speakers, making it more natural and reliable in scenarios with dynamic changes in people, such as meetings and social groups, thus broadening the application boundaries. Attached Figure Description

[0059] To more clearly illustrate the technical solutions in the embodiments of this application, the accompanying drawings used in the description of the embodiments will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0060] Figure 1 This is a flowchart of the method for actively predicting the next speaker in a multi-person dialogue scenario according to an embodiment of the present invention;

[0061] Figure 2This is a block diagram of the next speaker active prediction system in a multi-person dialogue scenario according to an embodiment of the present invention. Detailed Implementation

[0062] To enable those skilled in the art to better understand the present application, the technical solutions in the embodiments of the present application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. All other embodiments obtained by those skilled in the art based on the embodiments of the present application without creative effort are within the scope of protection of the present application.

[0063] In this application, the reference to "embodiment" means that a specific feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described in this application can be combined with other embodiments.

[0064] like Figure 1 As shown, this embodiment provides a method for proactively predicting the next speaker in a multi-person dialogue scenario, following a single-stage path of "context encoding → global semantic vector extraction → enhanced speaker embedding generation → matching score calculation → next speaker prediction," mainly including the following steps:

[0065] S1, Data Processing Stage, including:

[0066] S101: Dialogue data loading and formatting, specifically:

[0067] Multi-person dialogue data is read from a preprocessed JSONL file. Each dialogue contains multiple turns, and each turn includes speaker identifiers and corresponding statements. The dialogue data is divided into training, validation, and test sets to ensure the independence of dialogues in each set.

[0068] S102: Context window construction and sample generation, specifically:

[0069] For the i-th round (i ≥ 0) in each dialogue:

[0070] Define the context window size (For example, 8). Extract the first... The i-th round is used as the current context window.

[0071] Maintain the complete speaker position sequence simultaneously It stores the local IDs of all speakers from round 0 to round i, where the speaker IDs are encoded as 0, 1, 2, etc. in the order of their first appearance in the dialogue, and are used to calculate global long-term behavioral features;

[0072] All statements within the context window are concatenated in the [SPK{id}] Utterance format and separated by [SEP] to form the model's input text.

[0073] The speaker in the (i+1)th round is used as the prediction target. .like If it appears in the current context window, its label is the corresponding local ID, which is specifically assigned as... , This indicates the number of people who have spoken within the window. Otherwise, the label will be uniformly set to the new speaker identifier. Or a fixed value, such as .

[0074] S103: Candidate speaker set and dynamic feature construction, specifically:

[0075] For each sample, the present invention processes it in the following manner:

[0076] Candidate set construction: Extract all unique historical speakers from the start of the dialogue up to the current round, and form a set. In addition, a generalized identifier is added. This identifier is used to represent all speakers who did not appear in the conversation history. This ultimately forms the candidate set.

[0077] Dynamic temporal characteristics within the window: for historical speakers Calculate the relative position of the speaker's latest speech within the context window. Specifically, the position is calculated by counting backwards from the end of the window, setting the most recent speech to 1 and incrementing sequentially for the furthest speakers. For new speakers... Set relative position to , indicating "has not yet appeared".

[0078] Global long-term behavioral characteristics (8 dimensions): based on the complete historical speaker records up to the current round i. (i.e., rounds 0 to ith) the following 8-dimensional statistical feature vector is calculated in real time for each candidate speaker. This feature construction process strictly adheres to the principle of using only historical data to ensure no data leakage. For the new speaker identifier [NEW], its long-term behavioral features are fixed at [0,1,0,1,0,0,0.5]. The features of each dimension are defined as follows:

[0079] (1) Global speaking frequency: refers to the speaker's speaking frequency in the historical record. The ratio of the total number of occurrences in a given number to the total number of historical rounds, i.e.:

[0080]

[0081] This feature quantifies the speaker's overall engagement in the conversation.

[0082] (2) Normalized nearest speaker distance: Suppose the speaker is in The last round index in the middle is The nearest speech distance is... (If the speaker does not appear in history, then let) The distance is The normalized distance is... The larger the value, the longer the speaker remains silent.

[0083] (3) Local First Speaker: A binary feature indicating whether the speaker is the initiator of the current context window dialogue. If the speaker is the 0th position in the context window... If the value equals the speaker, the value is 1.0; otherwise, it is 0.0. This feature is used to capture the facilitator of the current topic.

[0084] (4) Candidate Status Identifier: This is a discrete numerical feature used to indicate the presence status and information completeness of a candidate speaker in the current dialogue. The value rules are as follows: For candidates who appear in both the history and the current context window, the value is 0.0. Such candidates possess both complete long-term behavioral gains and dynamic temporal gains. For candidates who appear in the history but not in the current context window (i.e., returners), the value is 0.5. Such candidates possess long-term behavioral gains but lack window-based dynamic temporal gains. This value suggests that the model should primarily rely on its historical profile for prediction. The value corresponding to the generalized new speaker identifier [NEW] is 1.0. Such candidates have neither specific historical behavior nor window position information. This value serves as a strong signal, indicating that the model prediction points to the open set category.

[0085] (5) Normalized average turn length: Calculate the average number of words (or characters) in all the speaker's historical speech, then divide by a preset normalization factor (e.g., 50.0), and truncate the result to the interval [0,1]. This feature reflects the speaker's usual speaking length.

[0086] (6) Dominance of Speech: This feature calculates the proportion of the total number of words (or phrases) spoken by a speaker in the historical dialogue to the total number of words spoken by all participants. Feature (1) "Global Speech Frequency" calculates the "proportion of the number of times spoken"; while this feature calculates the "proportion of the amount of content". This can effectively distinguish between "frequent echoers" (many times, low dominance) and "core speakers" (may have fewer times, but many words, high dominance), thereby eliminating the calculation redundancy with feature (1).

[0087] (7) Recent responsiveness is a binary feature that determines whether the speaker spoke in the round immediately preceding the current round i (i.e., the i-1th round). If so, the value is 1.0, indicating that the speaker has just participated in the conversation and is in an active state; otherwise, it is 0.0. This feature captures short-term interaction patterns.

[0088] (8) Consistency of speech: This measures the stability of the length of a speaker's historical speeches. Let the length sequence of all their historical speeches be denoted as . Then the consistency calculation is as follows

[0089]

[0090] in This represents the standard deviation. The closer this value is to 1, the more stable the speech length; when there are fewer than two historical speeches, the default value is 0.5. This feature is used to capture the stability of a speaker's speaking style.

[0091] This invention abandons the two-stage cascaded architecture of "detect first, then identify," and adopts an end-to-end single-stage design. All candidate speakers (i.e., the set of historical speakers appearing in the context window) With a generalized identifier representing a new speaker The comparison is performed within a unified feature space. The output type is directly applied to the entire candidate set. The probability distribution on:

[0092]

[0093] This design avoids error propagation caused by errors in the front-end detection module and enables the recognition of new speakers to directly benefit from comparative learning with historical speakers.

[0094] All calculations in this embodiment are strictly based on index In this section, the causality of features is ensured, and the use of future information is avoided. For historical speaker candidates that have never appeared in the history (appearing in the window but not in earlier history), their 1st, 2nd, 5th, 6th, 7th, and 8th dimension features will be calculated based on available data (possibly 0 or default values), while the 3rd and 4th dimensions will be 0. For the new speaker identifier [NEW], its 8-dimensional feature vector is fixed at [0,1,0,1,0,0,0.5]. This setting has a clear physical meaning: no historical frequency (0), furthest distance (1), not the first speaker (0), is a new speaker (1), no average length (0), no dominance (0), no recent response (0), default consistency (0.5).

[0095] S2. Model Structure Design and Forward Computation: The model described in this invention uses a pre-trained language model based on Transformer (such as RoBERTa-base) as the core context encoder, and adds an enhanced speaker dynamic representation generation module, including multi-head attention and gating mechanisms. The forward computation process is as follows:

[0096] S21: Contextual semantic encoding, specifically:

[0097] The input text ([SPK0] Utterance1 [SEP] [SPK1] Utterance2...) generated in step S102 is input into the pre-trained language model encoder. The encoder output is then retrieved. The hidden state corresponding to the marker is used as a global semantic gain. (H is the dimension of the hidden layer, which is 768 dimensions in the example), reflecting the real-time semantic features of the current dialogue.

[0098] S22: Enhance speaker dynamic representation.

[0099] The main purpose is for each candidate speaker Generate enhanced representations Specifically, it includes the following sub-steps:

[0100] Dynamic character buff generation: This will generate candidate... The 8-dimensional long-term behavioral features are input into a two-layer MLP (containing ReLU, LayerNorm, Dropout, and Tanh activations), and the output is a dynamic role gain. This process does not use any speaker ID embedding and is entirely driven by behavioral features.

[0101] Dynamic temporal gain embedding: Using a learnable positional embedding matrix, candidate... The relative position values ​​are mapped to vectors, and then passed through an MLP enhancement module to output dynamic timing gain. .

[0102] Basic gain feature fusion includes:

[0103] (1) Dynamic character buffs With dynamic timing gain The basic gain features are spliced ​​together and fused through a linear fusion layer. .

[0104] (2) All candidates Stacked as a sequence, input to a multi-head attention module (e.g., 8 heads), allowing candidate speaker features to interact, capturing the competition and cooperation relationships among candidates, and outputting attention-enhanced features. .

[0105] (3) Dynamic fusion through gating network (sigmoid) and To obtain enhanced basic features .

[0106] Context-aware deep fusion: Integrating global semantic gains , ,and The data is concatenated and fed into another MLP (Context-Aware Fusion Network). After a nonlinear transformation, the final enhanced speaker dynamic representation is output. .

[0107] S23: Matching score calculation and prediction, specifically:

[0108] calculate With each candidate representation The dot product similarity between them is used as the matching score, as follows:

[0109]

[0110] The Softmax function is applied to the scores of all candidates to obtain a probability distribution, and the candidate with the highest probability is selected as the prediction result. Specifically:

[0111]

[0112] In summary, this embodiment employs an enhanced speaker dynamic representation generation mechanism based on multi-source gain fusion. This mechanism generates a context-aware dynamic feature vector for each candidate speaker. (H is the hidden layer dimension), which fuses three independent information gains:

[0113] Long-term behavioral gains Based on the complete dialogue history up to the current moment, the 8-dimensional statistical features are calculated in real time and mapped by a multilayer perceptron (MLP) to represent the speaker's functional role in the global dialogue.

[0114] Dynamic timing gain : Through a learnable positional embedding layer, candidate speakers are placed in the current context window (e.g., most recent). The relative speaking position in a round is mapped to a vector, representing its recent active state.

[0115] Global semantic gain The [CLS] tag hidden state, obtained by encoding the context window (using a pre-trained model such as RoBERTa), reflects the immediate semantics and topic direction of the current dialogue.

[0116] The aforementioned gains are deeply fused through a multi-stage nonlinear fusion network incorporating multi-head attention and a gating mechanism, ultimately outputting a highly discriminative enhanced speaker representation. This provides robots with detailed participant profiles.

[0117] S3. Model training and optimization strategies, specifically:

[0118] S301: Adaptive Asymmetric Focus Loss Function

[0119] To effectively alleviate the class imbalance problem caused by the extreme sparsity of new speaker samples, this invention proposes an Adaptive Asymmetric Focal Loss (AFFL) for open set prediction tasks. This loss function, based on the standard focal loss, introduces a dynamic weighting mechanism, an asymmetric focusing strategy, and label smoothing regularization. For a training sample, the model outputs a logits vector of... The real label is The corresponding prediction probability is

[0120] First, calculate the labeled smoothing base cross-entropy loss.

[0121]

[0122] Among them, The label smoothing coefficient is set (preferably 0.1). Then, a dynamic weighting factor is defined. With dynamic focus index ;

[0123] Define the sample confidence metric as: This invention introduces an asymmetric parameterization mechanism to make the weighting factors... With focus index All are confidence levels The linear interpolation function is as follows:

[0124]

[0125]

[0126] The parameter settings satisfy and The asymmetric relationship. This enhances attention to low-confidence samples (typically new speakers). In a preferred embodiment, the following can be configured: , , , This allows for better results. These parameters can also be set as learning parameters constrained to positive values ​​via the Softplus function, achieving full adaptability.

[0127] The final loss function is:

[0128]

[0129] in:

[0130] The predicted probability of the true category;

[0131] For label smoothing ( Cross-entropy loss;

[0132] Parameter settings: ;

[0133] high Significantly amplify the new speaker (low) The gradient contribution of ).

[0134] This loss function uses higher-order focusing terms. Significantly amplify the new speaker (usually manifested as low) The gradient of the label is smoothed to suppress overfitting, so that the model can significantly improve the recall, F1 score and AUC-ROC index of new speakers while maintaining high accuracy for historical speakers.

[0135] S302: Optimizer and Learning Rate Scheduling, specifically:

[0136] The AdamW optimizer was used, and differentiated learning rates were set for different parameter groups in the model: a lower learning rate (1.2e-5) was used for the RoBERTa encoder parameters to stabilize semantic extraction capabilities; a higher learning rate (3e-5) was set for the parameters of newly added modules to accelerate the convergence of the feature fusion network. A linear learning rate decay strategy with preheating was adopted, with the preheating stage accounting for 10% of the total training steps.

[0137] S303: Class balancing and oversampling, specifically:

[0138] During training data loading, samples containing new speakers are oversampled to increase their proportion in the training set to approximately 20%, thus mitigating class imbalance.

[0139] S304: Early stopping and model selection, specifically:

[0140] Monitor Top-1 accuracy on the validation set; if it is continuous... A cycle (e.g.) If no improvement is observed, training will be terminated early, and the parameters of the best-performing model will be saved.

[0141] To verify the effectiveness of the method of this invention, a systematic comparative experiment was conducted on the MultiLight multi-person dialogue dataset. MultiLight is a large-scale open-domain role-playing dialogue dataset, and its test set contains 11,005 samples, of which 10,222 are from historical speakers and 783 are from new speakers, showing a significant class imbalance.

[0142] Two approaches were selected for comparison in the experiment: one used the large-scale language model Llama3-3B, and the experiment was conducted solely through cue word case learning (without parameter fine-tuning); the other involved fully supervised fine-tuning of the Llama3-3B model. The performance of these two approaches was compared with the single-stage end-to-end prediction model based on RoBERTa-base proposed in this invention. Evaluation metrics included global accuracy, new speaker F1 score, and mean reciprocal rank (MRR), used to measure overall prediction performance, new speaker recognition ability, and prediction ranking quality, respectively. The experimental results are shown in Table 1.

[0143] Table 1

[0144] index Llama3-3B (Few-Shot) Llama3-3B (Full Parameter Fine-Tuning) Method of the present invention Global accuracy 0.2810 0.5562 0.5815 New speaker's F1 score 0.3213 0.7619 0.8007 Mean Recess Rank (MRR) 0.3687 0.7471 0.7772 Parameters 3B 3B 125M

[0145] This invention achieves a significant lead in the key metric of new speaker recognition, which is crucial for robots to adapt to dynamic dialogues. Simultaneously, the model is lightweight with extremely low prediction latency, fully meeting the requirements for real-time robot interaction (typically <100ms). In contrast, while large model solutions offer acceptable accuracy, their enormous computational overhead and latency make them difficult to deploy in real-world robotic systems.

[0146] In a more specific reasoning case, let's take an intelligent meeting assistant robot as an example to illustrate the implementation process of this invention. Assume the context window size... The following is a sample dialogue:

[0147] Turn 1: [SPK0] "Hello everyone, we're here today to discuss project progress."

[0148] Turn 2: [SPK1] "My development is 80% complete."

[0149] Turn 3: [SPK0] "How is the testing team ready?"

[0150] Target prediction: The speaker for the next round. Current round index i=3 (predicting the speaker for the 4th round).

[0151] The specific processing flow for the above reasoning case is as follows:

[0152] Step 1: Audio Processing

[0153] The robot's ASR module converts the above dialogue into text. It identifies the speaker's identity through voice features, constructs a speaker profile, and compares the content of the speech with the next speaker prediction module to determine whether the speaker should be the robot itself.

[0154] Step 2: Construct the input text:

[0155] Extract the current round and its preceding rounds. The rounds serve as context windows, namely Turn 1 to Turn 3. Concatenate the rounds within the window in the format [SPK{id}] {utterance}, separated by [SEP]: Construct the input text: [SPK0] Hello everyone… [SEP] [SPK1] I’m here… [SEP] [SPK0] The testing team…

[0156] Step 3: Construct a set of candidate speakers And calculate the features:

[0157] Candidate speaker set Build: From the current context window Extract all unique local IDs of historical speakers to obtain a set. Add the new speaker identifier [NEW] to form the final candidate set. .

[0158] Dynamic time-series features (relative position within the window) calculation: SPK0 last appeared in Turn3, and its relative position is 1 according to the definition; SPK1 last appeared in Turn2, and its relative position is 2; [NEW]'s position is... Therefore, the relative position feature vector is

[0159] Long-term behavioral characteristics calculation: Calculate the feature vector matrix according to the definition in S103.

[0160] Step 4: Forward computation and prediction of the model:

[0161] Contextual encoding: The input text is fed into a pre-trained encoder (such as RoBERTa), and the hidden states marked with [CLS] are extracted as global semantic vectors. .

[0162] Enhanced speaker representation generation: for candidate set Each member in the algorithm, using its calculated long-term behavioral features and relative position features, generates a corresponding enhanced representation through the enhanced speaker dynamic representation generation module of this invention (including MLP, multi-head attention, gating fusion, etc.). .

[0163] Match score calculation: Calculate the global semantic vector Dot product similarity with each enhanced representation.

[0164] Probability Distribution and Prediction: Applying the Softmax function to the similarity scores yields a probability distribution. Assume the model output is: That is, the probability of SPK0 is 0.10, the probability of SPK1 is 0.26, and the probability of the new speaker [NEW] is 0.64.

[0165] Output: Select the candidate with the highest probability as the prediction result: This means the model predicts the next speaker will be a new participant not currently present in the context window. This result aligns with the scenario's intuition: after SPK0 (the project manager) asks about the "test team," a new test lead (SPK2) is likely to respond. Therefore, the bot avoids responding directly, preventing a poor conversational experience. If the prediction is that the bot represents the speaker, then the bot will proceed with subsequent dialogue generation and speech synthesis processes. Furthermore, in addition to these functions, the bot can preload relevant resources based on the general speaking characteristics of the [NEW] speaker to improve response speed and quality.

[0166] The above case clearly demonstrates the complete process from the initial dialogue to the final prediction, covering candidate set construction, two types of feature calculation, and the key steps of how the model integrates information to make decisions, as well as how the robot processes the output signal. Through proactive prediction, the robot is no longer a passive recorder but becomes a forward-looking meeting collaborator. It anticipates the natural direction of the conversation and makes multiple preparations in advance, making the final human-computer interaction smoother, more natural, and more efficient.

[0167] like Figure 2As shown, in another embodiment of this application, a next speaker active prediction system is provided in a multi-person dialogue scenario. This system is integrated into a multi-person dialogue robot system as a core module of dialogue decision-making. The system includes an audio acquisition unit, a speech recognition unit, a prediction processing unit, a speech judgment unit, a natural language generation unit, a speech synthesis unit, and a speech playback unit.

[0168] The audio acquisition unit uses a microphone array to acquire real-time audio signals in the dialogue scenario;

[0169] The speech recognition unit receives the audio signal output by the audio acquisition unit, converts it into a real-time text stream, and transmits it to the prediction processing unit;

[0170] The prediction processing unit receives the real-time text stream and performs active prediction of the next speaker based on the active prediction method for the next speaker in the multi-person dialogue scenario.

[0171] The speaking judgment unit receives the probability distribution output by the prediction processing unit and determines whether the next speaker is a robot character;

[0172] If the speech determination unit determines that the next speaker is a robot, the natural language generation unit receives the determination result, generates a text response, and transmits it to the speech synthesis unit.

[0173] The speech synthesis unit receives the text response, converts it into a speech signal, and transmits it to the speech playback unit;

[0174] The voice playback unit receives the voice signal and plays the voice through a speaker.

[0175] In another embodiment, an electronic device is provided for an active prediction method for the next speaker in a multi-person dialogue scenario. The electronic device may include a first processor, a first memory, and a bus, and may also include a computer program stored in the first memory and executable on the first processor, such as an active prediction program for the next speaker in a multi-person dialogue scenario.

[0176] Those skilled in the art will understand that all or part of the processes in the methods of the above embodiments can be implemented by a computer program instructing related hardware. The program can be stored in a non-volatile computer-readable storage medium, and when executed, it can include the processes of the embodiments of the above methods. Any references to memory, storage, databases, or other media used in the embodiments provided in this application can include non-volatile and / or volatile memory. Non-volatile memory can include read-only memory (ROM), programmable ROM (PROM), electrically programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), or flash memory. Volatile memory can include random access memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in various forms, such as static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), dual data rate SDRAM (DDRSDRAM), enhanced SDRAM (ESDRAM), synchronous link DRAM (SLDRAM), RAMbus direct RAM (RDRAM), direct memory bus dynamic RAM (DRDRAM), and RAMbus dynamic RAM (RDRAM), etc.

[0177] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0178] The above embodiments are preferred embodiments of the present invention, but the embodiments of the present invention are not limited to the above embodiments. Any changes, modifications, substitutions, combinations, or simplifications made without departing from the spirit and principle of the present invention shall be considered equivalent substitutions and shall be included within the protection scope of the present invention.

Claims

1. A method for proactively predicting the next speaker in a multi-person dialogue scenario, characterized in that, Includes the following steps: S1. Load formatted multi-person dialogue data, construct context windows and training samples, generate a set of candidate speakers and calculate the dynamic temporal features and global long-term behavioral features of each candidate. S2. A pre-trained language model based on Transformer is used as a context encoder. Multi-source gain fusion is used to generate enhanced speaker dynamic representations. The matching scores between the global semantic vector and each candidate representation are calculated and the prediction results are output. S3. An adaptive asymmetric focus loss function is used to alleviate the class imbalance problem, and the model training is completed by combining differentiated learning rate scheduling, oversampling and early stopping strategies.

2. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 1, characterized in that, Step S1 specifically includes: S101: Read multi-person dialogue data from a JSONL format file, where each dialogue contains multiple rounds of statements with speaker identifiers; S102: Define the size of the context window, extract the corresponding round range as the current context window, maintain the complete speaker's speaking position sequence, splice the sentences in the window to form the model input text, and use the next round speaker as the prediction target; S103: Construct a candidate speaker set, which includes all unique historical speakers up to the current round and generalized identifiers representing new speakers; calculate the relative positions of each candidate within the window as dynamic temporal features, and calculate multidimensional global long-term behavioral features based on the complete speaker speaking position sequence.

3. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 2, characterized in that, Multidimensional global long-term behavioral characteristics include: global speaking frequency, normalized nearest speaker distance, local first speaker, candidate status identifier, normalized average turn length, speaking dominance, recent responsiveness, and speaking consistency; Wherein, the global speaking frequency is the ratio of the total number of times a speaker appears in the complete speaker speaking position sequence to the total number of historical rounds; The normalized nearest speaking distance is the normalized result of the difference between the current round and the speaker's last speaking round relative to the length of the complete speaker speaking position sequence; The local first speaker is a binary feature that indicates whether the speaker is the first speaker in the current context window; The candidate status identifier is a discrete numerical feature used to indicate the candidate's existence status in the historical record and the current window, including three values: active historical state, regressive latent state, and unknown new ecosystem. The normalized average turn length is the normalized result of the average number of words in all of the speaker's historical speech relative to a preset normalization factor; The speaking dominance is the proportion of a speaker’s total historical speaking content to the total content of all speakers, and the content is calculated based on the number of words or phrases. The recent responsiveness is a binary feature used to determine whether the speaker spoke in the previous round immediately preceding the current round. The consistency of speech is calculated based on the standard deviation of the speaker's historical speech length sequence and is used to reflect the stability of the speaker's speech length.

4. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 1, characterized in that, The multi-source gain fusion described in step S2 includes the deep fusion of long-term behavioral gain, dynamic temporal gain, and global semantic gain. The specific process is as follows: S21: Input the input text into the pre-trained language model encoder and extract the hidden state corresponding to the [CLS] tag as the global semantic gain; S22: Input multidimensional long-term behavioral features into a two-layer MLP and output dynamic role gain; map relative position values ​​through a learnable position embedding matrix and output dynamic temporal gain; S23: The basic gain feature is obtained by splicing dynamic role gain and dynamic temporal gain and passing it through a linear fusion layer. The attention enhancement feature is output through a multi-head attention module. The attention enhancement feature and the basic gain feature are dynamically fused through a gating network to obtain the enhanced basic feature. S24: Concatenate global semantic gain, enhanced basic features and attention-enhanced features, and output the final enhanced speaker dynamic representation through a context-aware fusion network.

5. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 1, characterized in that, The prediction result in step S2 is obtained in the following way: The dot product similarity between the global semantic vector and each candidate representation is calculated as the matching score. The Softmax function is applied to all candidate scores to obtain the probability distribution, and the candidate with the highest probability is selected as the prediction result.

6. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 1, characterized in that, The adaptive asymmetric focus loss function described in step S3 is constructed based on the label-smoothed basic cross-entropy loss, and introduces a dynamic weighting factor and a dynamic focus index. Both the dynamic weighting factor and the dynamic focus index are linear interpolation functions of the true class prediction probability. By setting asymmetric parameter relationships, the focus on low-confidence samples is enhanced.

7. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 6, characterized in that, Final loss function for: in, The predicted probability of the true class. As a weighting factor, To focus on the index, The basic cross-entropy loss is labeled and smoothed. The calculation is as follows: in, The label smoothing coefficient. Indicates a candidate speaker. This represents the set of candidate speakers.

8. The method for actively predicting the next speaker in a multi-person dialogue scenario according to claim 1, characterized in that, The differentiated learning rate scheduling in step S3 specifically involves: using a lower learning rate for the pre-trained language model encoder parameters and a higher learning rate for the newly added module parameters, employing a linear learning rate decay strategy with preheating.

9. A system for proactively predicting the next speaker in a multi-person dialogue scenario, used to implement the method for proactively predicting the next speaker in a multi-person dialogue scenario as described in any one of claims 1-8, characterized in that, include: An audio acquisition unit is used to acquire real-time audio signals in a dialogue scenario using a microphone array. The speech recognition unit is used to receive the audio signal output by the audio acquisition unit, convert it into a real-time text stream and transmit it to the electronic device for active prediction of the next speaker; The speech judgment unit is used to receive the probability distribution of the predicted output and determine whether the next speaker is a robot character; The natural language generation unit is used to generate a text response when the next speaker is determined to be a robot. A speech synthesis unit is used to convert the text response into a speech signal; a speech playback unit is used to receive the speech signal and play it through a speaker.

10. An electronic device, characterized in that, The electronic device includes: At least one processor; and, A memory communicatively connected to the at least one processor; wherein, The memory stores computer program instructions that can be executed by the at least one processor to enable the at least one processor to perform the next speaker active prediction method in a multi-person dialogue scenario as described in any one of claims 1-8.

Images