Dialog processing method and apparatus, electronic device, and storage medium

Abstract

The present application discloses a dialog processing method and apparatus, an electronic device, and a storage medium. The method comprises: acquiring image information and text information corresponding to each turn of a multi-turn dialog, wherein the image information corresponding to each turn covers a plurality of participants; on the basis of the image information corresponding to each turn, determining a first probability that the plurality of participants are speakers of each turn; on the basis of the text information corresponding to each turn, determining a second probability that every two turns of the multi-turn dialog are from the same speaker; and on the basis of the first probability and the second probability, determining speakers of each turn. In embodiments of the present application, in a multi-turn multi-participant dialog, speaker prediction can be completed by using visual modal information and text modal information, and the introduction of rich visual modal information is beneficial to improving the accuracy of speaker prediction.

Description

Dialogue processing methods, devices, electronic equipment and storage media

[0001] This application claims priority to Chinese Patent Application No. 202410791941.5, filed on June 18, 2024, entitled "Dialogue Processing Method, Apparatus, Electronic Device and Storage Medium", the entire contents of which are incorporated herein by reference. Technical Field

[0002] This application relates to the field of artificial intelligence (AI) technology, and more particularly to a dialogue processing method, apparatus, electronic device, and storage medium. Background Technology

[0003] Multi-party dialogue refers to conversations involving more than two speakers. Compared to two-party dialogues, multi-party dialogues are more complex and diverse due to the variety of speakers and receivers. Research on multi-party dialogue mainly includes three sub-tasks: speaker prediction, utterance prediction, and addressee prediction. Existing research on multi-party dialogue primarily uses information from the text modality. However, this single-text modality approach ignores the rich information in real-world dialogue scenarios. Considering the complexity of speakers or receivers changing in each round of dialogue, relying solely on text modality information for speaker or receiver prediction leads to low prediction accuracy. Summary of the Invention

[0004] This application provides a dialogue processing method, apparatus, electronic device, and storage medium that can predict the speaker in multi-turn, multi-party dialogues using visual modal information and text modal information. The introduction of rich visual modal information helps to improve the accuracy of speaker prediction.

[0005] Firstly, embodiments of this application provide a dialogue processing method, applied to an electronic device or a chip within an electronic device. Taking the application of this method to an electronic device as an example, the method includes:

[0006] Acquire image and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants;

[0007] Based on the image information corresponding to each round of dialogue, determine the first probability that multiple participants are speakers in each round of dialogue;

[0008] Based on the text information corresponding to each round of dialogue, determine the second probability that every two rounds of dialogue in a multi-round dialogue come from the same speaker;

[0009] The speaker for each round of dialogue is determined based on the first probability and the second probability.

[0010] It can be seen that in multi-turn, multi-party dialogues, the electronic device can determine the first probability that multiple participants are speakers in each round of dialogue based on the image information corresponding to each round. Based on the text information corresponding to each round of dialogue, it can determine the second probability that every two rounds of dialogue come from the same speaker. Finally, the speaker prediction result is obtained based on the first and second probabilities. Since the first probability is obtained based on visual modality information, determining the speaker in each round of dialogue based on the first and second probabilities adds visual modality information to the text modality information. The introduction of rich visual modality information helps to improve the accuracy of speaker prediction.

[0011] In one possible implementation, based on the image information corresponding to each round of dialogue, the first probability of multiple participants being speakers in each round of dialogue is determined, including:

[0012] In the case where the image information corresponding to each round of dialogue is a video with speech, for any round of dialogue in a multi-round dialogue, the matching degree between the facial features and speech features of each person in the video corresponding to that round of dialogue is obtained; where the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the facial features and speech features of each person.

[0013] When the image information corresponding to each round of dialogue is an image, for any round of dialogue in a multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability of the participant corresponding to each face speaking.

[0014] In this implementation, when the image information corresponding to each round of dialogue is a video with audio, the electronic device can calculate the matching degree between the features of each face in the video and the audio features to obtain the first probability that multiple participants are the speakers in each round of dialogue. When the image information corresponding to each round of dialogue is an image, the electronic device can obtain the probability that each participant corresponding to each face is speaking through the features of each face, thereby obtaining the first probability that multiple participants are the speakers in each round of dialogue. That is, regardless of whether it is a video context or an image context, the electronic device can perform speaker recognition through rich visual information.

[0015] In one possible implementation, based on the text information corresponding to each round of dialogue, a second probability is determined that every two rounds of dialogue come from the same speaker, including:

[0016] Obtain the text features of the text information corresponding to each round of dialogue;

[0017] Obtain the difference in text features between every two rounds of dialogue;

[0018] Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined.

[0019] The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

[0020] In this implementation, the electronic device determines the similarity between the text features corresponding to each two rounds of dialogue based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, thereby obtaining the second probability that each two rounds of dialogue come from the same speaker, which can then provide usable text modal information for subsequent speaker recognition optimization tasks.

[0021] In one possible implementation, the speaker for each round of dialogue is determined based on a first probability and a second probability, including:

[0022] The speaker for each round of dialogue is determined by solving matrix X in a quadratic programming problem; the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB stX∈{0,1} m×l

[0023] Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant out of l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probability, B be the matrix composed of the first probability, and α be the hyperparameter for adjusting the weights of A and B.

[0024] In this implementation, the electronic device can find a matrix X that maximizes the objective function so that the speaker's prediction results can meet the outputs of the visual module and the text module as much as possible, thereby improving the accuracy of the speaker's prediction.

[0025] In one possible implementation, multi-turn dialogue consists of m turns.

[0026] The method also includes:

[0027] A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue.

[0028] The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

[0029] In this implementation, a prompt is constructed using image information corresponding to each round of dialogue, the speaker in each round, and text information corresponding to rounds 1 to (m-1). This prompt is then used to fine-tune the neural network model, resulting in a dialogue content prediction model capable of capturing speaker information, thus improving the prediction accuracy. In application, this dialogue content prediction model can predict speech content or provide dialogue responses based on image information, speaker information, and contextual content within the dialogue scene.

[0030] Secondly, embodiments of this application provide a dialogue processing apparatus, applied to an electronic device or a chip in an electronic device; the apparatus includes an acquisition unit and a processing unit;

[0031] The acquisition unit is used to acquire image and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants;

[0032] The processing unit is used to determine, based on the image information corresponding to each round of dialogue, a first probability that multiple participants are speakers in each round of dialogue; based on the text information corresponding to each round of dialogue, a second probability that every two rounds of dialogue in a multi-round dialogue come from the same speaker; and to determine the speaker in each round of dialogue based on the first probability and the second probability.

[0033] In one possible implementation, regarding determining the first probability that multiple participants are speakers in each round of dialogue based on the image information corresponding to each round of dialogue, the processing unit is specifically used for:

[0034] In the case where the image information corresponding to each round of dialogue is a video with speech, for any round of dialogue in a multi-round dialogue, the matching degree between the facial features and speech features of each person in the video corresponding to that round of dialogue is obtained; where the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the facial features and speech features of each person.

[0035] When the image information corresponding to each round of dialogue is an image, for any round of dialogue in a multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability of the participant corresponding to each face speaking.

[0036] In one possible implementation, regarding the second probability of determining that two rounds of dialogue originate from the same speaker based on the text information corresponding to each round of dialogue, the processing unit is specifically used for:

[0037] Obtain the text features of the text information corresponding to each round of dialogue;

[0038] Obtain the difference in text features between every two rounds of dialogue;

[0039] Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined.

[0040] The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

[0041] In one possible implementation, the processing unit is specifically used to determine the speaker for each round of dialogue based on a first probability and a second probability:

[0042] The speaker for each round of dialogue is determined by solving matrix X in a quadratic programming problem; the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB stX∈{0,1} m×l

[0043] Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant out of l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probability, B be the matrix composed of the first probability, and α be the hyperparameter for adjusting the weights of A and B.

[0044] In one possible implementation, the multi-turn dialogue is an m-turn dialogue; the processing unit is also used for:

[0045] A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue.

[0046] The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

[0047] It should be understood that since the method embodiments and the device embodiments are different presentations of the same technical concept, the content of the first aspect of the embodiments of this application should be adapted to the second aspect of the embodiments of this application simultaneously, and can achieve the same or similar beneficial effects, which will not be repeated here.

[0048] Thirdly, embodiments of this application provide an electronic device including a processor, a memory, a communication interface, and one or more programs, the one or more programs being stored in the memory and configured to, when executed by the processor, cooperate with the communication interface to implement the method in any of the embodiments of the first aspect described above.

[0049] Fourthly, embodiments of this application provide a chip, including: a processor, configured to call and run a computer program from a memory, causing a device on which the chip is installed to perform the method as described in any of the embodiments of the first aspect above.

[0050] Fifthly, embodiments of this application provide a computer-readable storage medium storing a computer program for execution by a device, wherein the computer program, when executed, implements the method as described in any of the embodiments of the first aspect above.

[0051] Sixthly, embodiments of this application provide a computer program product that, when run by a device, causes the device to perform the method as described in any of the embodiments of the first aspect above. Attached Figure Description

[0052] To more clearly illustrate the technical solutions in the embodiments of this application or the background art, the accompanying drawings used in the embodiments of this application or the background art will be described below.

[0053] Figure 1 is a schematic diagram of a system architecture provided in an embodiment of this application;

[0054] Figure 2 is a flowchart illustrating a dialogue processing method provided in an embodiment of this application;

[0055] Figure 3 is a schematic diagram of a speaker prediction model provided in an embodiment of this application;

[0056] Figure 4 is a flowchart illustrating another dialogue processing method provided in an embodiment of this application;

[0057] Figure 5 is a schematic diagram of another system architecture provided in an embodiment of this application;

[0058] Figure 6 is a schematic diagram of a dialogue processing device provided in an embodiment of this application;

[0059] Figure 7 is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. Detailed Implementation

[0060] The terms "first," "second," "third," and "fourth," etc., used in the specification, claims, and accompanying drawings of this application are used to distinguish different objects, not to describe a specific order. Furthermore, the terms "comprising" and "having," and any variations thereof, are intended to cover non-exclusive inclusion. For example, a process, method, system, product, or apparatus that includes a series of steps or units is not limited to the listed steps or units, but may optionally include steps or units not listed, or may optionally include other steps or units inherent to these processes, methods, products, or apparatuses.

[0061] In this document, the term "embodiment" means that a particular feature, structure, or characteristic described in connection with an embodiment may be included in at least one embodiment of this application. The appearance of this phrase in various places throughout the specification does not necessarily refer to the same embodiment, nor is it a mutually exclusive, independent, or alternative embodiment. It will be explicitly and implicitly understood by those skilled in the art that the embodiments described herein can be combined with other embodiments.

[0062] The terms “component,” “module,” “system,” etc., used in this specification are used to refer to computer-related entities, hardware, firmware, combinations of hardware and software, software, or software in execution. For example, a component can be, but is not limited to, a process running on a processor, a processor, an object, an executable file, an execution thread, a program, and / or a computer. As illustrated, applications running on electronic devices and electronic devices themselves can be components. One or more components may reside in a process and / or an execution thread, and components may be located on a single computer and / or distributed among two or more computers. Furthermore, these components can be executed from various computer-readable media on which various data structures are stored. Components can communicate, for example, via local and / or remote processes based on signals having one or more data packets (e.g., data from two components interacting with another component between a local system, a distributed system, and / or a network, such as the Internet interacting with other systems via signals).

[0063] First, a brief introduction to the relevant terms and technical background used in this application will be provided to facilitate understanding by those skilled in the art.

[0064] (1) Machine learning system: Based on input data and labels, the parameters of the machine learning model are trained through optimization methods such as gradient descent, and the trained model is finally used to complete the prediction of unknown data.

[0065] (2) Multimodal dialogue: Multimodal dialogue system is an emerging artificial intelligence technology that can interact with users through various input methods (such as voice, text, images, etc.) and provide information and services to users through various output methods (such as voice, text, images, etc.).

[0066] (3) Fine-tuning: The process of continuing training using specific training samples based on the pre-trained model.

[0067] (4) Full fine-tuning: Fine-tuning in which all parameters of the model can be adjusted.

[0068] (5) Efficient fine-tuning: Fine-tuning where only some of the model's parameters can be adjusted. Compared to full-parameter fine-tuning, efficient fine-tuning updates fewer model parameters, trains faster, and can quickly adapt to the target domain and data. Common methods include Low-Rank Adaptation of Large Language Models (LoRA) and Prompt-Tuning.

[0069] (6) Large Language Model (LLM): A super-large deep learning model that is pre-trained with a large amount of data. It can not only generate natural language text, but also deeply understand the meaning of the text and handle various natural language tasks, such as text summarization, question answering, translation, etc.

[0070] (7) Prompt: In the field of artificial intelligence, this usually refers to a text input used to guide the model to generate corresponding outputs. This input is usually a short sentence or a piece of text, the purpose of which is to guide the model to generate a specific type of text or perform a specific task.

[0071] The Multimodal Emotion Lines Dataset (MELD) aims to serve speaker emotion recognition tasks in multimodal, multi-party dialogues. This dataset collects dialogues from movie or TV series clips, each containing both text transcription and audio recordings. This allows researchers to utilize different modal information (text and audio) for emotion recognition, contributing to the advancement of dialogue systems and emotional intelligence. Existing solutions using MELD for multi-party dialogue emotion recognition do not focus on dialogue tasks such as speaker identification and speech content prediction, and primarily utilize text content and audio information for emotion classification, ignoring the rich information contained in the visual modality.

[0072] One related technology proposes a multi-turn dialogue semantic understanding subsystem based on a multimodal emotion recognition system, including a data acquisition device, an output device, and an emotion analysis software system. The emotion analysis software system performs comprehensive analysis and reasoning on the data obtained by the data acquisition device and finally outputs the results to the output device. The emotion analysis software system includes a multi-turn dialogue semantic understanding subsystem. This subsystem adds an attention mechanism for emotion recognition to the input utterance of the current turn, building upon the traditional seq2seq language generation model. The dialogue management incorporates emotion tracking from previous turns of dialogue over time. Each utterance spoken by the current user is input into a bidirectional Long Short-Term Memory (LSTM) encoder. Then, the currently identified different emotion states are merged with the encoder output of the previously generated user utterance and input together into the decoder. This scheme, while performing semantic understanding of multi-turn dialogues, uses the user's utterance as the input to the LSTM for emotion recognition, and in essence, still does not consider visual modality information.

[0073] To overcome the shortcomings of existing technologies, embodiments of this application provide a dialogue processing method, apparatus, electronic device, and storage medium. This dialogue processing method can be implemented based on the system architecture shown in Figure 1. As shown in Figure 1, the system architecture includes a data collection device 101 (optional), a server 102, and a client device 103 (optional), all connected via a network. The data collection device 101 acquires the data (corpus, images, etc.) required by this solution through various means (manual input, network search, etc.) and transmits it to the server 102. The server 102 executes the relevant steps of the dialogue processing method provided by this solution to obtain a model capable of speaker prediction and dialogue content prediction. The model trained or fine-tuned by the server 102 is typically deployed on various servers for access or use by the client device 103. Optionally, the operation of the data collection device 101 can also be performed by the server 102.

[0074] The technical solution provided in this application will be described in detail below with reference to specific implementation methods.

[0075] Please refer to Figure 2, which is a flowchart illustrating a dialogue processing method provided in an embodiment of this application. This method can be implemented based on the system architecture shown in Figure 1. As shown in Figure 2, the method includes steps 201-204:

[0076] 201: Obtain the image and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants.

[0077] In this embodiment, each round of dialogue can be a single sentence spoken by one participant, multiple sentences spoken continuously by one participant around a topic (or question), or multiple sentences spoken non-continuously by one participant around a topic. The image information corresponding to each round of dialogue can be a video with audio captured during the speaker's speech or an image captured during the speaker's speech. For example, the multiple participants in the image information corresponding to each round of dialogue can be all participants in multiple rounds of dialogue, or only the participants in the current round of dialogue.

[0078] 202: Based on the image information corresponding to each round of dialogue, determine the first probability that multiple participants are speakers in each round of dialogue.

[0079] For example, this application provides a speaker prediction model, as shown in Figure 3. This speaker prediction model includes a visual module, a text module, and a binary optimization module. The visual module deploys an active speaker detection model and a convolutional neural network (CNN) model. When the image information corresponding to each round of dialogue is a video with speech, for any round of dialogue in a multi-round dialogue, the matching degree between the features of each face and the speech features in the video corresponding to that round of dialogue is obtained. The first probability that the participant corresponding to each face is the speaker in this round of dialogue is then the matching degree between the features of each face and the speech features. Specifically, the video corresponding to each round of dialogue is input into the speaker detection model, which extracts the features of each face and the speech features, and then calculates the matching degree. When the image information corresponding to each round of dialogue is an image, for any round of dialogue in a multi-round dialogue, the probability of each participant speaking (i.e., the probability of each participant speaking the current utterance) is obtained based on the features of each face in the image corresponding to that round of dialogue. Therefore, the first probability that each participant is the speaker in this round of dialogue is the probability that each participant is speaking. Thus, the output of the visual module in Figure 3 is a matrix composed of the first probabilities that multiple participants are the speakers in each round of dialogue.

[0080] For example, the speaker recognition model uses the TalkNet model as the matching model, and the CNN model uses the Inception model pre-trained on VGGFace2 (a dataset for face recognition). The Inception model is fine-tuned using the dataset provided in this application embodiment, thereby enabling it to determine whether a face is speaking.

[0081] It should be noted that if multiple participants in the image information corresponding to each round of dialogue are only participants in the current round of dialogue, then the first probability that other participants in multiple rounds of dialogue are speakers in the current round of dialogue is 0. For example, taking any round of dialogue as dialogue P, assuming that the image information corresponding to dialogue P includes participant 1, participant 2, and participant 3, and all participants in multiple rounds of dialogue also include participant 4 and participant 5, then the probability that participant 4 and participant 5 are speakers in dialogue P can be set to 0.

[0082] In this implementation, when the image information corresponding to each round of dialogue is a video with audio, the electronic device can calculate the matching degree between the features of each face in the video and the audio features to obtain the first probability that multiple participants are the speakers in each round of dialogue. When the image information corresponding to each round of dialogue is an image, the electronic device can obtain the probability that each participant corresponding to each face is speaking through the features of each face, thereby obtaining the first probability that multiple participants are the speakers in each round of dialogue. That is, regardless of whether it is a video context or an image context, the electronic device can perform speaker recognition through rich visual information.

[0083] 203: Based on the text information corresponding to each round of dialogue, determine the second probability that every two rounds of dialogue in a multi-round dialogue come from the same speaker.

[0084] In this embodiment, the text module in Figure 3 is used to comprehensively consider the entire dialogue history and determine the probability that each two rounds of dialogue were spoken by the same speaker. Specifically, the text module takes the text information of each round of dialogue as input, obtains the text features corresponding to the text information of each round of dialogue, and then obtains the difference between the text features corresponding to each two rounds of dialogue. Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined. This similarity can be calculated using the following formula: a ik =σ(W2GeLU(W1[h i h k ;|h i -h k |]+b1)+b2)

[0085] Among them, a ik The text feature h represents the text feature corresponding to the i-th round of dialogue. i The text feature h corresponding to the k-th round of dialogue k The similarity between them is defined by W1, W2, b1, b2, which are the parameters of the two linear layers, GeLU and σ, which are activation functions, and "||" indicates taking the absolute value.

[0086] The electronic device determines the second probability that the two rounds of dialogue come from the same speaker by the similarity between the text features corresponding to each pair of dialogues. Therefore, the output of the text module in Figure 3 is a matrix composed of the second probabilities that the two rounds of dialogue come from the same speaker in a multi-round dialogue.

[0087] For example, the text module deploys text encoding / decoding models such as Bidirectional Encoder Representations from Transformers (BERT), Robustly Optimized BERT Approach (RoBERTa), or Decoding-Enhanced BERT with Disentangled Attention (DeBERTa). For instance, the text module can use DeBERTa-v3 as a prediction model for every two rounds of dialogue from the same speaker. The DeBERTa-v3 model is fine-tuned using existing datasets (such as the publicly available Ubuntu Dialogue Corpus dataset for dialogue system research) and the dataset provided in this application's embodiments. Taking DeBERTa-v3 as an example, a start symbol is added before the text information of each round of dialogue. <bos>Then add <bos>The text information of the multi-turn dialogue (turn 1 to turn m) is concatenated and used as the input to the DeBERTa-v3 model. The DeBERTa-v3 model is based on the preceding text of each turn of dialogue. <bos>Symbols are used to extract text features corresponding to the text information of each round of dialogue. The text features corresponding to each two rounds of dialogue are concatenated with the absolute value of their difference. The concatenated features are processed by two linear layers to output the similarity between the text features corresponding to the two rounds of dialogue.

[0088] In this implementation, the electronic device determines the similarity between the text features corresponding to each two rounds of dialogue based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, thereby obtaining the second probability that each two rounds of dialogue come from the same speaker, which can then provide usable text modal information for subsequent speaker recognition optimization tasks.

[0089] 204: Determine the speaker for each round of dialogue based on the first probability and the second probability.

[0090] In this embodiment, the binary optimization module uses information provided by the visual module and the text module as input to assign a speaker to each round of dialogue. Specifically, it determines the speaker for each round of dialogue by constructing an objective function for a quadratic programming problem and solving for the matrix X in the objective function. The objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB

[0091] stX∈{0,1} m×l

[0092] Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant out of l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probabilities, B be the matrix composed of the first probabilities, and α be a hyperparameter for adjusting the weights of A and B. A∈R m×m , B∈R l×m R is the set of real numbers, and st is the constraint condition.

[0093] In this implementation, the electronic device can find a matrix X that maximizes the objective function so that the speaker's prediction results can meet the outputs of the visual module and the text module as much as possible, thereby improving the accuracy of the speaker's prediction.

[0094] To verify the performance of the speaker prediction scheme provided in this application, researchers collected a large amount of multimodal multi-party dialogue data from TV dramas as a dataset. As shown in Table 1, the dataset generally includes a subset with 5 rounds of iterative fine-tuning and a subset with 8 rounds of iterative fine-tuning. Each subset is further divided into three types: training set, test set (easy), and test set (difficult). Each type includes information such as the number of dialogues and the number of dialogue rounds. Based on this, corresponding datasets were constructed from two dimensions: video context and image context. For example, in the video context, the average number of faces in the video, the average duration of faces, the proportion of frames where the speaker is not present, and the proportion of all frames where the speaker is not present in the dialogue were also statistically analyzed. Similarly, in the image context, the average number of faces per frame, the proportion of frames where the speaker is not present, and the proportion of all frames where the speaker is not present in the dialogue were also statistically analyzed.

[0095] Table 1

[0096] In Table 1, the "Easy Test Set" refers to the test set where faces in the image information are annotated, while the "Hard Test Set" refers to the test set where face annotations are randomly removed from already annotated images. The "Easy Test Set" aims to fully utilize information from the visual modality, while the "Hard Test Set" aims to use information from the text modality to assist visual modality information in speaker prediction when visual modality information is insufficient.

[0097] The speaker recognition model and some baseline models (including Large Language and Vision Assistant (LLaVA), 3-shot Chat Generative Pre-trained Transformer (3-shot ChatGPT), generative multimodal model Emu, etc.) used in this application embodiment have achieved the following accuracy after fine-tuning on the dataset provided in this application embodiment: Table 2 shows the accuracy of these models.

[0098] Table 2

[0099] As can be seen from Table 2, the speaker recognition model provided in this application has better overall prediction accuracy than other baseline models in both image and video contexts.

[0100] It can be seen that in multi-turn, multi-party dialogues, the electronic device can determine the first probability that multiple participants are speakers in each round of dialogue based on the image information corresponding to each round. Based on the text information corresponding to each round of dialogue, it can determine the second probability that every two rounds of dialogue come from the same speaker. Finally, the speaker prediction result is obtained based on the first and second probabilities. Since the first probability is obtained based on visual modality information, determining the speaker in each round of dialogue based on the first and second probabilities adds visual modality information to the text modality information. The introduction of rich visual modality information helps to improve the accuracy of speaker prediction.

[0101] Please refer to Figure 4, which is a flowchart illustrating another dialogue processing method provided in this application embodiment. This method can be implemented based on the system architecture shown in Figure 1. As shown in Figure 4, the method includes steps 401-406:

[0102] 401: Obtain the image and text information corresponding to each round of dialogue in a multi-turn dialogue; the image information corresponding to each round of dialogue includes multiple participants; the multi-turn dialogue consists of m rounds.

[0103] 402: Based on the image information corresponding to each round of dialogue, determine the first probability that multiple participants are speakers in each round of dialogue.

[0104] 403: Based on the text information corresponding to each round of dialogue, determine the second probability that every two rounds of dialogue in a multi-round dialogue come from the same speaker.

[0105] 404: The speaker for each round of dialogue is determined based on the first probability and the second probability.

[0106] The specific implementation of steps 401-404 can correspond to the relevant description of steps 201-204 in the embodiment shown in FIG2, and can achieve the same or similar beneficial effects.

[0107] 405: Construct a prompt using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. Predict the prompt using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue.

[0108] In this embodiment, the image information corresponding to each round of dialogue is subjected to feature extraction using an image coding model such as Visual Transformer (ViT) to obtain the corresponding feature vector. The prompt constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue is as follows:

[0109] [Frame 1][Speaker 1][Dialogue Content 1][Frame 2][Speaker 2][Dialogue Content 2]……[Frame m][Speaker m]

[0110] Here, [frame m] represents the feature vector of the image information corresponding to the m-th round of dialogue. The constructed prompt is input into a pre-trained neural network model, which first converts [speaker] and [dialogue content] into tokens, and then combines the feature vector and tokens to predict and output the dialogue content of the m-th round of dialogue.

[0111] 406: Fine-tune the neural network model based on the text information corresponding to the m-th round of dialogue to obtain the dialogue content prediction model.

[0112] In this embodiment, the electronic device calculates the loss using the predicted dialogue content of the m-th round of dialogue and the actual content of the m-th round of dialogue. Based on the loss value of the loss function, the neural network model is fine-tuned, and a trained dialogue content prediction model is obtained after multiple iterations.

[0113] This application implements a unimodal dialogue model without image input and a multimodal dialogue model with image input on the dataset given in Table 1. The unimodal dialogue model uses Large Language Model Meta AI (LLaMA) 2-7B as the pre-trained model, and the multimodal dialogue model uses Emu-14B as the pre-trained model.

[0114] To test the performance of the dialogue content prediction model, given multiple (multi-turn) candidate dialogues, the model selects the candidate dialogue with the lowest perplexity. Specifically, for any candidate dialogue C among the multiple candidate dialogues, assuming that dialogue C has m turns, the first n turns of dialogue C are input into the dialogue content prediction model. Then, a segment of content from any candidate dialogue among the multiple candidate dialogues is input into the dialogue content prediction model. The model calculates the probability that the given segment of content is the subsequent dialogue content of the first n turns of dialogue C, and uses this probability to calculate the perplexity of the given segment of content. The test accuracy of the dialogue content prediction model with the LLaMA2-7B model as the pre-trained model and the dialogue content prediction model with the Emu-14B model as the pre-trained model are shown in Table 3.

[0115] Table 3

[0116] The input to the dialogue content prediction model using the LLaMA2-7B model as the pre-trained model does not include image information, while the input to the dialogue content prediction model using the Emu-14B model as the pre-trained model does include image information. As shown in Table 3, the dialogue content prediction model obtained by fine-tuning the scheme provided in this application is closer to the accuracy of a given real speaker in terms of prediction accuracy, and it verifies that the speaker information provided by the speaker recognition model helps the dialogue content prediction model to predict dialogue content.

[0117] As shown in Table 3, the speaker recognition model and dialogue content prediction model constructed by the dialogue processing method provided in this application embodiment are also applicable to scenarios without speaker information, such as users who have not yet registered with the system. For example, a new user (unknown speaker) entering a public place, smart cockpit, or smart home scenario. By constructing the input prompt of the model using the corresponding unknown speaker label, and inputting it into the dialogue content prediction model finely tuned by this application embodiment for inference, a dialogue response can be completed.

[0118] To facilitate a more comprehensive understanding of the dialogue processing method proposed in Figure 4, this application embodiment also provides a system architecture, as shown in Figure 5. This system architecture mainly includes a speaker recognition module and a dialogue content prediction module. The speaker recognition module includes a vision module, a text module, and a binary optimization module (shown by dashed lines). The dialogue content prediction module includes prompt construction and an LLM (shown by dashed lines), for example, the LLM could be LoRA. The speaker recognition module takes text and image information from multiple rounds of dialogue as input, and the speaker information predicted by the speaker recognition module is used as input to the dialogue content prediction module. The image information can be video or pictures. The specific implementations of the vision module, text module, and binary optimization module can be found in the relevant descriptions in the embodiment shown in Figure 3. The prompt construction uses the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue to construct the prompt. The prompt serves as input to the LLM, and the LLM outputs the dialogue content of the m-th round of dialogue. Therefore, the output of the entire system architecture includes speaker information and the predicted dialogue content. The dialogue content prediction module is fine-tuned to enable it to capture different speakers.

[0119] The speaker recognition module and the dialogue content prediction module are used to implement the decoupled two-stage scheme provided in the embodiments of this application. The speaker recognition module consists of three decoupled sub-modules: the visual module, the text module, and the binary optimization module.

[0120] In this implementation, a prompt is constructed using image information corresponding to each round of dialogue, the speaker in each round, and text information corresponding to rounds 1 to (m-1). This prompt is then used to fine-tune the neural network model, resulting in a dialogue content prediction model capable of capturing speaker information, thus improving the prediction accuracy. In application, this dialogue content prediction model can predict speech content or provide dialogue responses based on image information, speaker information, and contextual content within the dialogue scene.

[0121] Please refer to Figure 6, which is a schematic diagram of a dialogue processing device provided in an embodiment of this application. This device can be applied to an electronic device or a chip in an electronic device. As shown in Figure 6, the device may include an acquisition unit 601 and a processing unit 602; wherein:

[0122] The acquisition unit 601 is used to acquire image information and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants;

[0123] The processing unit 602 is used to determine, based on the image information corresponding to each round of dialogue, a first probability that multiple participants are speakers in each round of dialogue; based on the text information corresponding to each round of dialogue, a second probability that every two rounds of dialogue in the multi-round dialogue come from the same speaker; and to determine the speaker in each round of dialogue based on the first probability and the second probability.

[0124] As can be seen, in multi-round, multi-party dialogue, the device shown in Figure 6 can determine the first probability that multiple participants are speakers in each round of dialogue based on the image information corresponding to each round of dialogue. Based on the text information corresponding to each round of dialogue, it can determine the second probability that every two rounds of dialogue come from the same speaker. Finally, the speaker prediction result is obtained based on the first and second probabilities. Since the first probability is obtained based on visual modal information, determining the speaker in each round of dialogue based on the first and second probabilities means adding visual modal information to the text modal information. The introduction of rich visual modal information helps to improve the accuracy of speaker prediction.

[0125] In one possible implementation, in determining the first probability that multiple participants are speakers in each round of dialogue based on the image information corresponding to each round of dialogue, the processing unit 602 is specifically used for:

[0126] In the case where the image information corresponding to each round of dialogue is a video with speech, for any round of dialogue in a multi-round dialogue, the matching degree between the facial features and speech features of each person in the video corresponding to that round of dialogue is obtained; where the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the facial features and speech features of each person.

[0127] When the image information corresponding to each round of dialogue is an image, for any round of dialogue in a multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability of the participant corresponding to each face speaking.

[0128] In one possible implementation, regarding the second probability of determining that two rounds of dialogue come from the same speaker based on the text information corresponding to each round of dialogue, the processing unit 602 is specifically used for:

[0129] Obtain the text features of the text information corresponding to each round of dialogue;

[0130] Obtain the difference in text features between every two rounds of dialogue;

[0131] Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined.

[0132] The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

[0133] In one possible implementation, in determining the speaker for each round of dialogue based on a first probability and a second probability, the processing unit 602 is specifically configured to:

[0134] The speaker for each round of dialogue is determined by solving matrix X in a quadratic programming problem; the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB

[0135] stX∈{0,1} m×l

[0136] Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant out of l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probability, B be the matrix composed of the first probability, and α be the hyperparameter for adjusting the weights of A and B.

[0137] In one possible implementation, the multi-turn dialogue is an m-turn dialogue; the processing unit is also used for:

[0138] A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue.

[0139] The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

[0140] It should be noted that the implementation of each unit described in FIG6 can also correspond to the descriptions of the embodiments shown in FIG2 to FIG5. Furthermore, the beneficial effects of the device described in FIG6 can be described in the corresponding descriptions of the embodiments shown in FIG2 to FIG5, and will not be repeated here.

[0141] Based on the description of the above method and device embodiments, this application also provides an electronic device. Please refer to FIG7, which is a schematic diagram of the structure of an electronic device provided in this application embodiment. The electronic device includes at least a processor 701, a memory 702, and a communication interface 703, which are interconnected via a bus 704. This electronic device can be used to execute relevant steps of a dialogue processing method. The electronic device can be a computer device such as a desktop computer, laptop computer, server, cloud server, or embedded device, for example, it can be the server 102 in FIG1. ​​The processor 701 in the electronic device is used to read the computer program code stored in the memory 702 and execute the method of any one of the embodiments shown in FIG2 to FIG5.

[0142] The memory 702 includes, but is not limited to, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM), or compact disc read-only memory (CD-ROM), and is used to store related computer programs and data.

[0143] Processor 701 can be one or more central processing units (CPUs). If processor 701 is a CPU, the CPU can be a single-core CPU or a multi-core CPU.

[0144] For example, the processor 701 in the electronic device can be used to read one or more programs stored in the memory 702 described above, and perform the following operations:

[0145] Acquire image and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants;

[0146] Based on the image information corresponding to each round of dialogue, determine the first probability that multiple participants are speakers in each round of dialogue;

[0147] Based on the text information corresponding to each round of dialogue, determine the second probability that every two rounds of dialogue in a multi-round dialogue come from the same speaker;

[0148] The speaker for each round of dialogue is determined based on the first probability and the second probability.

[0149] In one possible implementation, processor 701 performs a first probability determination based on image information corresponding to each round of dialogue, identifying multiple participants as speakers in each round of dialogue, including:

[0150] In the case where the image information corresponding to each round of dialogue is a video with speech, for any round of dialogue in a multi-round dialogue, the matching degree between the facial features and speech features of each person in the video corresponding to that round of dialogue is obtained; where the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the facial features and speech features of each person.

[0151] When the image information corresponding to each round of dialogue is an image, for any round of dialogue in a multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability of the participant corresponding to each face speaking.

[0152] In one possible implementation, processor 701 performs a second probability determination based on the text information corresponding to each round of dialogue, including:

[0153] Obtain the text features of the text information corresponding to each round of dialogue;

[0154] Obtain the difference in text features between every two rounds of dialogue;

[0155] Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined.

[0156] The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

[0157] In one possible implementation, processor 701 performs the function of determining the speaker for each round of dialogue based on a first probability and a second probability, including:

[0158] The speaker for each round of dialogue is determined by solving matrix X in a quadratic programming problem; the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB stX∈{0,1} m×l

[0159] Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant out of l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probability, B be the matrix composed of the first probability, and α be the hyperparameter for adjusting the weights of A and B.

[0160] In one possible implementation, multi-turn dialogue consists of m turns.

[0161] Processor 701 is also used for:

[0162] A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue.

[0163] The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

[0164] It should be noted that the implementation of each operation can also correspond to the description of the method in any of the embodiments shown in Figures 2 to 5.

[0165] It should be noted that although the electronic device shown in Figure 7 only illustrates the processor 701, memory 702, communication interface 703, and bus 704, those skilled in the art should understand that in specific implementations, the electronic device may also include other devices necessary for normal operation. Furthermore, depending on specific needs, those skilled in the art should understand that the electronic device may also include hardware devices for implementing other additional functions. Moreover, those skilled in the art should understand that the electronic device may only include the devices necessary for implementing the embodiments of this application, and not necessarily all the devices shown in Figure 7.

[0166] This application also provides a chip, including: a processor for calling and running a computer program from a memory, causing a device with the chip installed to perform the method described in any of the embodiments shown in Figures 2 to 5 above. This chip can be a chip in an electronic device.

[0167] This application also provides a computer-readable storage medium (memory) storing a computer program that, when executed, implements the method described in any of the embodiments shown in Figures 2 to 5. It is understood that the computer-readable storage medium here can include both built-in storage media within a device and extended storage media supported by the device. The computer-readable storage medium provides storage space containing the device's operating system. Furthermore, one or more computer programs suitable for loading and execution by the device's processor are also stored in this storage space. It should be noted that the computer-readable storage medium here can be high-speed RAM or non-volatile memory, such as at least one disk storage device; optionally, it can also be at least one computer-readable storage medium located remotely from the aforementioned processor.

[0168] This application also provides a computer program product, which includes computer program code. When the computer program code is run by an electronic device, the method flow described in any one of the embodiments in Figures 2 to 5 is implemented.

[0169] In the above embodiments, the descriptions of each embodiment have different focuses. For parts not described in detail in a certain embodiment, please refer to the relevant descriptions in other embodiments.

[0170] It should be understood that the processor mentioned in the embodiments of this application can be a CPU, or other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0171] It should also be understood that the memory mentioned in the embodiments of this application can be volatile memory or non-volatile memory, or may include both volatile and non-volatile memory. Non-volatile memory can be ROM, Programmable Read-Only Memory (PROM), EPROM, Electrically Erasable Programmable Read-Only Memory (EEPROM), or flash memory. Volatile memory can be RAM, which is used as an external cache. By way of example, but not limitation, many forms of RAM are available, such as Static Random Access Memory (SRAM), Dynamic Random Access Memory (DRAM), Synchronous Dynamic Random Access Memory (SDRAM), Double Data Rate Synchronous Dynamic Random Access Memory (DDR SDRAM), Enhanced Synchronous Dynamic Random Access Memory (ESDRAM), Synchlink Dynamic Random Access Memory (SLDRAM), and Direct Rambus RAM (DR RAM).

[0172] It should be noted that when the processor is a general-purpose processor, DSP, ASIC, FPGA, or other programmable logic device, discrete gate or transistor logic device, or discrete hardware component, the memory (storage module) is integrated into the processor.

[0173] It should be noted that the memories described herein are intended to include, but are not limited to, these and any other suitable types of memories.

[0174] It should be understood that in the various embodiments of this application, the order of the above-mentioned processes does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of this application.

[0175] In the several embodiments provided in this application, it should be understood that the disclosed systems, apparatuses, and methods can be implemented in other ways. For example, the apparatus embodiments described above are merely exemplary. For instance, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. For example, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Furthermore, the coupling or direct coupling or communication connection shown or discussed may be through some interfaces; the indirect coupling or communication connection between apparatuses or units may be electrical, mechanical, or other forms.

[0176] The units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; that is, they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment according to actual needs.

[0177] Furthermore, the functional units in the various embodiments of this application can be integrated into one processing unit, or each unit can exist physically separately, or two or more units can be integrated into one unit. If the integrated unit is implemented as a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium.

[0178] In this application, "at least one" means one or more, and "more" means two or more. "And / or" describes the relationship between related objects, indicating that three relationships can exist. For example, A and / or B can represent: A alone, A and B simultaneously, or B alone, where A and B can be singular or plural. In the textual description of this application, the character " / " generally indicates that the preceding and following related objects have an "or" relationship.

[0179] The steps in the method of this application embodiment can be adjusted, combined, or deleted according to actual needs.

[0180] The modules in the device of this application embodiment can be merged, divided, and deleted according to actual needs.

[0181] The above-described embodiments are only used to illustrate the technical solutions of this application, and are not intended to limit it. Although this application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some of the technical features. Such modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of this application.< / bos> < / bos> < / bos>

Claims

1. A dialogue processing method, characterized in that, The method includes: Acquire image and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants; Based on the image information corresponding to each round of dialogue, a first probability is determined that the multiple participants are the speakers in each round of dialogue; Based on the text information corresponding to each round of dialogue, a second probability is determined that every two rounds of dialogue in the multi-round dialogue come from the same speaker. The speaker for each round of dialogue is determined based on the first probability and the second probability.

2. The method according to claim 1, characterized in that, The step of determining the first probability that the multiple participants are speakers in each round of dialogue based on the image information corresponding to each round of dialogue includes: When the image information corresponding to each round of dialogue is a video with audio, for any round of dialogue in the multi-round dialogue, the matching degree between the features of each face and the audio features in the video corresponding to that round of dialogue is obtained; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the features of each face and the audio features. When the image information corresponding to each round of dialogue is an image, for any round of dialogue in the multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to the any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability that the participant corresponding to each face speaking.

3. The method according to claim 1 or 2, characterized in that, The step of determining the second probability that every two rounds of dialogue come from the same speaker based on the text information corresponding to each round of dialogue includes: Obtain the text features of the text information corresponding to each round of dialogue; Obtain the difference in text features corresponding to every two rounds of dialogue; Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined. The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

4. The method according to any one of claims 1-3, characterized in that, Determining the speaker for each round of dialogue based on the first probability and the second probability includes: The speaker for each round of dialogue is determined by solving matrix X in the quadratic programming problem; wherein the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB s.t.X∈{0,1} m×l Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant among l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probabilities, B be the matrix composed of the first probabilities, and α be the hyperparameter for adjusting the weights of A and B.

5. The method according to any one of claims 1-3, characterized in that, The multi-turn dialogue is an m-turn dialogue; The method further includes: A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue. The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

6. A dialogue processing device, characterized in that, The device includes an acquisition unit and a processing unit; The acquisition unit is used to acquire image information and text information corresponding to each round of dialogue in a multi-round dialogue; the image information corresponding to each round of dialogue includes multiple participants; The processing unit is used to determine, based on the image information corresponding to each round of dialogue, the first probability that the plurality of participants are speakers in each round of dialogue; Based on the text information corresponding to each round of dialogue, a second probability is determined that every two rounds of dialogue in the multi-round dialogue come from the same speaker; the speaker of each round of dialogue is determined based on the first probability and the second probability.

7. The apparatus according to claim 6, characterized in that, In determining the first probability that the plurality of participants are speakers in each round of dialogue based on the image information corresponding to each round of dialogue, the processing unit is specifically used for: When the image information corresponding to each round of dialogue is a video with audio, for any round of dialogue in the multi-round dialogue, the matching degree between the features of each face and the audio features in the video corresponding to that round of dialogue is obtained; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the matching degree between the features of each face and the audio features. When the image information corresponding to each round of dialogue is an image, for any round of dialogue in the multi-round dialogue, the probability of the participant corresponding to each face speaking is obtained based on the features of each face in the image corresponding to the any round of dialogue; wherein, the first probability that the participant corresponding to each face is the speaker in this round of dialogue is the probability that the participant corresponding to each face speaking.

8. The apparatus according to claim 6 or 7, characterized in that, In determining the second probability that two rounds of dialogue originate from the same speaker based on the text information corresponding to each round of dialogue, the processing unit is specifically used for: Obtain the text features of the text information corresponding to each round of dialogue; Obtain the difference in text features corresponding to every two rounds of dialogue; Based on the text features corresponding to each two rounds of dialogue and the difference between the text features corresponding to each two rounds of dialogue, the similarity between the text features corresponding to each two rounds of dialogue is determined. The similarity between the text features corresponding to every two rounds of dialogue is determined as the second probability.

9. The apparatus according to any one of claims 6-8, characterized in that, In determining the speaker for each round of dialogue based on the first probability and the second probability, the processing unit is specifically configured to: The speaker for each round of dialogue is determined by solving matrix X in the quadratic programming problem; wherein the objective function of the quadratic programming problem satisfies: Maximizef(X)=(1-α)X T AX+αXB s.t.X∈{0,1} m×l Where Maximize represents maximizing the objective function f(X), X ij X represents the probability that the j-th participant among l participants is the speaker in the i-th round of the m-round dialogue; l is the number of participants, and m is the number of rounds of dialogue; T Let X be the transpose of X, A be the matrix composed of the second probabilities, B be the matrix composed of the first probabilities, and α be the hyperparameter for adjusting the weights of A and B.

10. The apparatus according to any one of claims 6-8, characterized in that, The multi-turn dialogue is an m-turn dialogue; the processing unit is also used for: A prompt is constructed using the image information corresponding to each round of dialogue, the speaker of each round of dialogue, and the text information corresponding to the first to (m-1)th rounds of dialogue. The prompt is then predicted using a pre-trained neural network model to obtain the text information corresponding to the mth round of dialogue. The neural network model is fine-tuned based on the text information corresponding to the m-th round of dialogue to obtain a dialogue content prediction model.

11. An electronic device, characterized in that, It includes a processor, a memory, a communication interface, and one or more programs, said one or more programs being stored in the memory and configured to, when executed by the processor, cooperate with the communication interface to implement the method as described in any one of claims 1-5.

12. A computer-readable storage medium, characterized in that, The computer-readable storage medium stores a computer program for execution by the device, which, when executed, implements the method of any one of claims 1-5.

13. A computer program product, characterized in that, When the computer program product is run by the device, the device performs the method as described in any one of claims 1-5.

Images