A method, apparatus and equipment for processing multimodal generative dialogue tasks

Abstract

This application discloses a multimodal generative dialogue task processing method, apparatus, and device, including: receiving input text content, visual content, and streaming voice question content; when the full-duplex detection head in the target generative dialogue task processing model detects that the streaming voice question content has been received, encoding the streaming voice question content using an audio encoder, and converting the encoded audio feature vector into a first lexical unit in a preset unified semantic space using an audio adapter; encoding the text content using a text encoder to obtain a second lexical unit in the preset unified semantic space; encoding the visual content using a visual encoder, and converting the encoded image feature vector into a third lexical unit in the preset unified semantic space using a visual adapter; and predicting and generating response content based on the first, second, and third lexical units. This application improves the voice pause detection effect and increases the accuracy of the response content.

Description

Technical Field

[0001] This application relates to the field of artificial intelligence technology, and in particular to a method, apparatus, device, and computer-readable storage medium for processing multimodal generative dialogue tasks. Background Technology

[0002] Humans perceive the world through multiple sensory modalities (such as hearing, vision, and language). Therefore, developing models that can simulate these multimodal perceptual abilities is key to achieving Artificial General Intelligence (AGI). In recent years, Large Language Models (LLMs) have demonstrated outstanding performance in text understanding and reasoning, spurring research into multimodal capabilities such as visual-language alignment and instruction following, leading to the development of Multimodal Large Language Models (MLLMs). These models, by introducing visual encoders and cross-modal alignment mechanisms, have initially achieved joint text-image understanding. However, in complex visual reasoning tasks (such as long video analysis), they remain susceptible to the heterogeneity of different modalities, resulting in limited reasoning performance.

[0003] While these works have validated the feasibility of audio fusion, limitations remain. Current multimodal generative dialogue task processing models involve pre-training with images or videos, followed by multimodal alignment and audio expansion. This requires handling extremely long visual sequences and massive amounts of pre-training data, resulting in significant computational and time consumption and high training costs. Existing models' full-duplex interruption functionality heavily relies on an external voice activity detection module and a full-duplex stop detection module, which are somewhat disconnected from the model itself and produce poor speech stop detection results. Most current multimodal models support mixed image-text input or mixed audio-text input. Limited intermodal contextual information transmission leads to partial understanding and inaccurate responses.

[0004] In summary, effectively addressing the problems of high training costs, poor speech stop detection, one-sided understanding, and inaccurate responses in current multimodal generative dialogue task processing models is a pressing issue that needs to be resolved by those skilled in the art. Summary of the Invention

[0005] The purpose of this application is to provide a multimodal generative dialogue task processing method that improves speech stop detection, reduces model training costs, enhances the understanding ability of the target generative dialogue task processing model, and improves the accuracy of response content. Another purpose of this application is to provide a multimodal generative dialogue task processing method, apparatus, device, and computer-readable storage medium.

[0006] To solve the above-mentioned technical problems, this application provides the following technical solution:

[0007] It receives input text content, visual content, and streaming voice questions;

[0008] When the full-duplex detection head in the target generative dialogue task processing model detects that the receiving of the streaming voice question content is complete, the audio encoder in the target generative dialogue task processing model encodes the streaming voice question content, and the audio adapter converts the encoded audio feature vector into the first word element of the preset unified semantic space.

[0009] The text content is encoded using the text encoder in the target generative dialogue task processing model to obtain the second word element of the preset unified semantic space;

[0010] The visual content is encoded using the visual encoder in the target generative dialogue task processing model, and the encoded image feature vector is converted into a third word in the preset unified semantic space using the visual adapter.

[0011] The target generative dialogue task processing model is used to predict and generate response content based on the first word, the second word, and the third word to complete this multimodal generative dialogue task processing.

[0012] In one specific embodiment of this application, a training process for the target generative dialogue task processing model is also included, the training process of the target generative dialogue task processing model comprising:

[0013] The initial generative dialogue task processing model is trained with speech alignment to obtain the first generative dialogue task processing model with the input speech and input text aligned.

[0014] The first generative dialogue task processing model is fine-tuned for speech understanding to obtain the second generative dialogue task processing model.

[0015] The second generative dialogue task processing model is fine-tuned in terms of speech output to obtain the target generative dialogue task processing model.

[0016] In one specific embodiment of this application, speech alignment training is performed on the initial generative dialogue task processing model, including:

[0017] Obtain each speech-text sample pair;

[0018] For each speech-text sample pair, the speech vector is processed using the audio encoder and audio adapter in the initial generative dialogue task processing model to obtain the first vector of the preset unified semantic space, and the text vector of each speech-text sample pair is obtained as the second vector in the preset unified semantic space.

[0019] For each first vector, calculate the first similarity between the first vector and each second vector;

[0020] For each second vector, calculate the second similarity between the second vector and each first vector;

[0021] Calculate the bidirectional contrastive learning loss based on each first similarity score and each second similarity score;

[0022] The initial generative dialogue task processing model is iteratively trained based on the bidirectional contrastive learning loss.

[0023] In one specific embodiment of this application, fine-tuning the speech understanding of the first generative dialogue task processing model includes:

[0024] For each position in the preset unified semantic space of the response content sequence of preset length, the prefix and trimodal representation are input into the first generative dialogue task processing model during prediction to obtain the conditional probability of the prediction result of the position.

[0025] Calculate the prediction error loss based on the conditional probability corresponding to each position;

[0026] The first generative dialogue task processing model is iteratively trained based on the prediction error loss.

[0027] In one specific embodiment of this application, the target generative dialogue task processing model is used to predict and generate response content based on the first lexical unit, the second lexical unit, and the third lexical unit, including:

[0028] The target generative dialogue task processing model is used to predict and generate speech response content and text response content based on the first word, the second word, and the third word.

[0029] In one specific embodiment of this application, fine-tuning the speech output of the second generative dialogue task processing model includes:

[0030] Obtain the cross-entropy loss of the correct prediction results generated at each position by the second generative dialogue task processing model given a multimodal context;

[0031] Obtain the actual Mel spectrum frames corresponding to the voice response content samples;

[0032] Calculate the first norm of the voice response content sample and the real Mel spectrum frame;

[0033] Obtain the predicted duration of the corresponding speech unit of the speech response content sample, and obtain the actual duration of the speech unit corresponding to the actual Mel spectrum frame; wherein, the speech unit is a frame or a phoneme;

[0034] Calculate the squared Euclidean norms of the predicted duration and the actual duration;

[0035] The second generative dialogue task processing model is iteratively trained based on the cross-entropy loss, the first-order norm, and the squared Euclidean norm.

[0036] In one specific embodiment of this application, a detection process for the completion of receiving the streaming voice question content is also included. This detection process includes:

[0037] When the duration of the silence state detected by the full-duplex detection head in the target generative dialogue task processing model reaches a preset silence duration threshold, it is determined that the reception of the streaming voice question content is complete.

[0038] A multimodal generative dialogue task processing device, comprising:

[0039] The content receiving module is used to receive input text content, visual content, and streaming voice questions.

[0040] The first word element acquisition module is used to encode the streaming voice question content using the audio encoder in the target generative dialogue task processing model when the full-duplex detection head in the target generative dialogue task processing model detects that the receiving of the streaming voice question content has been completed, and to convert the encoded audio feature vector into the first word element of the preset unified semantic space using the audio adapter.

[0041] The second lexical unit acquisition module is used to encode the text content using the text encoder in the target generative dialogue task processing model to obtain the second lexical unit of the preset unified semantic space.

[0042] The third lexical unit acquisition module is used to encode the visual content using the visual encoder in the target generative dialogue task processing model, and to convert the encoded image feature vector into the third lexical unit of the preset unified semantic space using the visual adapter.

[0043] The generative dialogue task processing module is used to predict and generate response content based on the first word, the second word, and the third word using the target generative dialogue task processing model, so as to complete this multimodal generative dialogue task processing.

[0044] A multimodal generative dialogue task processing device, comprising:

[0045] Memory, used to store computer programs;

[0046] A processor is used to implement the steps of the multimodal generative dialogue task processing method as described above when executing the computer program.

[0047] A computer-readable storage medium storing a computer program that, when executed by a processor, implements the steps of the multimodal generative dialogue task processing method described above.

[0048] The multimodal generative dialogue task processing method provided in this application receives input text content, visual content, and streaming speech question content. When the full-duplex detection head in the target generative dialogue task processing model detects that the streaming speech question content has been received, the audio encoder in the target generative dialogue task processing model encodes the streaming speech question content, and the audio adapter converts the encoded audio feature vector into a first word in a preset unified semantic space. The text encoder in the target generative dialogue task processing model encodes the text content to obtain a second word in the preset unified semantic space. The visual encoder in the target generative dialogue task processing model encodes the visual content, and the visual adapter converts the encoded image feature vector into a third word in the preset unified semantic space. The target generative dialogue task processing model predicts and generates response content based on the first, second, and third words to complete the multimodal generative dialogue task processing.

[0049] As can be seen from the above technical solutions, by embedding the full-duplex detection head into the model, the full-duplex stop detection module is integrated with the model. Through the multimodal fusion representation of input text, audio, and images, it provides more comprehensive input information than traditional full-duplex stop detection modules. This allows the model to determine in real-time whether to interrupt or wait for user input during the decoding stage, significantly reducing latency and false positive rates, and improving the speech stop detection effect. By mapping image feature vectors, audio feature vectors, and text feature vectors to a pre-defined unified semantic space, it achieves temporal consistency parsing of long texts, long videos, and real-time speech, realizing end-to-end mixed modal understanding and reducing model training costs. The unified representation method enables mixed modal contextual memory, improving the understanding ability of the target generative dialogue task processing model and increasing the accuracy of response content.

[0050] Accordingly, this application also provides a multimodal generative dialogue task processing apparatus, device, and computer-readable storage medium corresponding to the above-mentioned multimodal generative dialogue task processing method, which have the above-mentioned technical effects, and will not be repeated here. Attached Figure Description

[0051] To more clearly illustrate the technical solutions in the embodiments or related technologies of this application, the accompanying drawings used in the description of the embodiments or related technologies will be briefly introduced below. Obviously, the accompanying drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0052] Figure 1 This is a flowchart illustrating one implementation of the multimodal generative dialogue task processing method in this application.

[0053] Figure 2 This is an architecture diagram of a generative dialogue task processing model in an embodiment of this application;

[0054] Figure 3 This is a flowchart illustrating the implementation of another multimodal generative dialogue task processing method in this application embodiment;

[0055] Figure 4 This is a diagram illustrating the three-stage training process of a generative dialogue task processing model in an embodiment of this application.

[0056] Figure 5 This is a structural block diagram of a multimodal generative dialogue task processing device according to an embodiment of this application;

[0057] Figure 6 This is a structural block diagram of a multimodal generative dialogue task processing device according to an embodiment of this application;

[0058] Figure 7 This is a schematic diagram of the specific structure of a multimodal generative dialogue task processing device provided in an embodiment of this application. Detailed Implementation

[0059] To enable those skilled in the art to better understand the present application, the present application will be further described in detail below with reference to the accompanying drawings and specific embodiments. Obviously, the described embodiments are merely some embodiments of the present application, and not all embodiments. Based on the embodiments in this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0060] See Figure 1 , Figure 1This is a flowchart of an implementation of a multimodal generative dialogue task processing method in this application, which may include the following steps.

[0061] S101: Receives input text content, visual content, and streaming voice question content.

[0062] When processing the current generative dialogue task, the input to the target generative dialogue task processing model can include multimodal components consisting of text content, visual content, and streaming speech questions. The model receives input text content, visual content, and streaming speech questions.

[0063] Visual content can include images and videos, while streaming speech questions can be questions posed through successive outputs of speech. The question object can be visual content, such as information contained in an image; it can also be text content, such as information contained in a table; or it can be content previously learned by the model that is unrelated to the currently input text or visual content.

[0064] S102: When the full-duplex detection head in the target generative dialogue task processing model detects that the streaming voice question content has been received, the audio encoder in the target generative dialogue task processing model encodes the streaming voice question content, and the audio adapter converts the encoded audio feature vector into the first word of the preset unified semantic space.

[0065] The target generative dialogue task processing model includes a full-duplex detection head for detecting the reception status of streaming voice questions. During the reception of streaming voice questions, the full-duplex detection head in the target generative dialogue task processing model is used to detect whether the reception of the streaming voice questions has been completed.

[0066] See Figure 2 , Figure 2 This is an architecture diagram of a generative dialogue task processing model according to an embodiment of this application. The target generative dialogue task processing model includes an audio encoder and an audio adapter. The audio encoder is responsible for mapping speech features to the semantic space of a large multimodal language model, and the audio decoder converts the semantic code back to speech. The audio encoder, through pre-training, replaces the self-attention mechanism and one-dimensional convolution (Conv1D) module with causal mechanisms, thereby accepting streaming audio input and reducing inference latency. The audio decoder is a 6-layer decoding Transformer used to synthesize the final audio.

[0067] When the full-duplex detection head in the target generative dialogue task processing model detects that the streaming voice question content has been received, the audio encoder in the target generative dialogue task processing model encodes the streaming voice question content, and the audio adapter converts the encoded audio feature vector into the first token in the preset unified semantic space.

[0068] The goal-oriented generative dialogue task processing model can accept any combination of input modalities from text, vision, and speech.

[0069] S103: Encode the text content using the text encoder in the target generative dialogue task processing model to obtain the second word element in the preset unified semantic space.

[0070] like Figure 2 As shown, the target generative dialogue task processing model also includes a text encoder. After receiving the input text content, the text encoder in the target generative dialogue task processing model encodes the text content to obtain the second word unit in the preset unified semantic space.

[0071] S104: The visual content is encoded using the visual encoder in the target generative dialogue task processing model, and the encoded image feature vector is transformed into a third word in a predefined unified semantic space using the visual adapter.

[0072] like Figure 2 As shown, the target generative dialogue task processing model also includes a visual encoder and a visual adapter. After receiving the input visual content, the visual encoder in the target generative dialogue task processing model encodes the visual content, and the visual adapter transforms the encoded image feature vector into a third word in a predefined unified semantic space.

[0073] Visual encoders can employ an improved Vision Transformer (ViT) architecture to support the processing of dynamic, high-resolution input images.

[0074] S105: Utilize the target generative dialogue task processing model to predict and generate response content based on the first, second, and third lexical units to complete this multimodal generative dialogue task processing.

[0075] After obtaining the first, second, and third lexical units in a predefined unified semantic space, the target generative dialogue task processing model predicts and generates response content based on the first, second, and third lexical units, thereby completing this multimodal generative dialogue task processing. This application's embodiment maps visual patches, audio tokens, and text tokens to a predefined unified semantic space, achieving time-consistent parsing of long texts, long videos, and real-time audio.

[0076] The generated response content can include both voice and text responses, thereby enabling cross-verification between the voice and text responses, improving the accuracy of the response content, and enhancing the user experience.

[0077] As can be seen from the above technical solutions, by embedding the full-duplex detection head into the model, the full-duplex stop detection module is integrated with the model. Through the multimodal fusion representation of input text, audio, and images, it provides more comprehensive input information than traditional full-duplex stop detection modules. This allows the model to determine in real-time whether to interrupt or wait for user input during the decoding stage, significantly reducing latency and false positive rates, and improving the speech stop detection effect. By mapping image feature vectors, audio feature vectors, and text feature vectors to a pre-defined unified semantic space, it achieves temporal consistency parsing of long texts, long videos, and real-time speech, realizing end-to-end mixed modal understanding and reducing model training costs. The unified representation method enables mixed modal contextual memory, improving the understanding ability of the target generative dialogue task processing model and increasing the accuracy of response content.

[0078] It should be noted that, based on the above embodiments, this application also provides corresponding improvement solutions. In subsequent embodiments, steps that are the same as or corresponding to those in the above embodiments can be referred to each other, and the corresponding beneficial effects can also be referred to each other. These improvements will not be elaborated upon in the following improved embodiments.

[0079] See Figure 3 , Figure 3 This is another implementation flowchart of the multimodal generative dialogue task processing method in the embodiments of this application. The method may include the following steps.

[0080] S301: Train the initial generative dialogue task processing model with speech alignment to obtain the first generative dialogue task processing model with input speech and input text aligned.

[0081] See Figure 4 , Figure 4This diagram illustrates the three-stage training process of a generative dialogue task processing model in an embodiment of this application. The initial generative dialogue task processing model is trained with speech alignment to obtain a first generative dialogue task processing model with input speech and input text aligned. The audio representation is then accurately mapped to the language semantic space by updating the audio adapter.

[0082] In one specific embodiment of this application, training the initial generative dialogue task processing model for speech alignment may include the following steps:

[0083] Step 1: Obtain each speech-text sample pair;

[0084] Step 2: For the speech vector in each speech-text sample pair, process it using the audio encoder and audio adapter in the initial generative dialogue task processing model to obtain the first vector in the preset unified semantic space, and obtain the second vector of the text vector in each speech-text sample pair in the preset unified semantic space.

[0085] Step 3: For each first vector, calculate the first similarity between the first vector and each second vector;

[0086] Step 4: For each second vector, calculate the second similarity between the second vector and each first vector;

[0087] Step 5: Calculate the bidirectional contrastive learning loss based on each first similarity score and each second similarity score;

[0088] Step 6: Iteratively train the initial generative dialogue task processing model based on the bidirectional contrastive learning loss.

[0089] For ease of description, the above six steps can be combined for explanation.

[0090] The initial generative dialogue task processing model is trained on speech alignment to obtain speech-text sample pairs. For the speech vector in each speech-text sample pair, the audio encoder and audio adapter in the initial generative dialogue task processing model are used for processing to obtain a first vector in a preset unified semantic space. A second vector in the preset unified semantic space is also obtained from the text vector in each speech-text sample pair. For each first vector, a first similarity is calculated between the first vector and each second vector. For each second vector, a second similarity is calculated between the second vector and each first vector. A bidirectional contrastive learning loss is calculated based on the first and second similarities. The initial generative dialogue task processing model is then iteratively trained based on the bidirectional contrastive learning loss. Bidirectional constraints prevent representation collapse and ensure the symmetry and consistency of the two modalities in the unified space, laying the foundation for subsequent multimodal fusion and generation.

[0091] The formula for calculating the loss during speech alignment training is as follows:

[0092] ;

[0093] in, It is a bidirectional contrastive learning loss used to bring speech vectors and text vectors to the same semantic coordinate system. Assume a mini-batch contains N pairs of samples; the i-th speech item is processed by an audio encoder and an audio adapter to obtain a vector. The corresponding vector in the latent space of the language model is the subtitle text that is aligned with it. Similarity Using a cosine form, while temperature The sharpness of the softmax distribution is controlled. The loss consists of two parts: one is fixed. Combine it with all texts in the same batch. Softmax is applied to the similarity scores, expecting only true matches. The probability is the highest; secondly, it is symmetrically fixed. Then with all voices To perform softmax, This represents the text modal representation of the j-th sample in the batch. Both terms are in the form of cross-entropy, and the final result is obtained by averaging over N pairs of samples. Intuitively, it brings correctly paired speech and text vectors closer together, while pushing incorrect pairings further apart using N-1 negative samples within a batch. The lower the temperature, the greater the required similarity gap.

[0094] S302: Fine-tune the first generative dialogue task processing model by performing speech understanding adjustments to obtain the second generative dialogue task processing model.

[0095] like Figure 4 As shown, after training the initial generative dialogue task processing model with speech alignment to obtain a first generative dialogue task processing model with input speech and input text aligned, the first generative dialogue task processing model is fine-tuned with speech understanding to obtain a second generative dialogue task processing model. A large amount of mixed image, text, and audio instruction data is introduced, and the three-modal backbone network, visual adapter, audio adapter, and audio decoder are unfrozen to complete multimodal instruction understanding and unified sequence modeling.

[0096] In one specific embodiment of this application, fine-tuning the speech understanding of the first generative dialogue task processing model may include the following steps:

[0097] Step 1: For each position in a predefined unified semantic space within a response content sequence of a predefined length, input both the prefix and the three-modal representation into the first generative dialogue task processing model during prediction to obtain the conditional probability of the position prediction result.

[0098] Step 2: Calculate the prediction error loss based on the conditional probability corresponding to each position;

[0099] Step 3: Iteratively train the first generative dialogue task processing model based on the prediction error loss.

[0100] For ease of description, the three steps above can be combined for explanation.

[0101] Fine-tuning the speech understanding of the first generative dialogue task processing model can include, for each position in a predefined unified semantic space in a response content sequence of a predefined length, inputting both the prefix and the three-modal representation into the first generative dialogue task processing model during prediction, obtaining the conditional probability of the prediction result for each position, calculating the prediction error loss based on the conditional probability corresponding to each position, and iteratively training the first generative dialogue task processing model based on the prediction error loss.

[0102] The formula for calculating the fine-tuning loss of speech understanding is as follows:

[0103] ;

[0104] In a target sequence of length T, the model predicts the result at position k. When, always put the prefix and three-modal characterization The data is fed into the network to obtain the conditional probability of the label. Taking the logarithm of all positions and adding a negative sign is equivalent to minimizing the negative log-likelihood of the target sequence, which is to maximize the probability that the model generates the correct output sequence under trimodal conditions; summing over k from 1 to T accumulates the prediction error of the entire sentence. Optimizing this loss can encourage the parameters to... Given a common context of audio, image, and text, as well as already generated prefixes, the system learns to continuously produce the next tag that conforms to the distribution of the training corpus, thereby completing the fine-tuning of semantic understanding and text generation capabilities for multimodal instructions.

[0105] S303: Fine-tune the speech output of the second generative dialogue task processing model to obtain the target generative dialogue task processing model.

[0106] After fine-tuning the first generative dialogue task processing model for speech understanding to obtain the second generative dialogue task processing model, fine-tuning the speech output of the second generative dialogue task processing model to obtain the target generative dialogue task processing model. By adding an audio decoder and jointly optimizing the audio decoder and language decoder end-to-end on a longer corpus of text with subtitles, the consistency and fluency of text generation and speech synthesis are ensured. By sequentially training the initial generative dialogue task processing model for speech alignment, fine-tuning speech understanding, and fine-tuning speech output, the target generative dialogue task processing model capable of end-to-end processing of multimodal generative dialogue tasks is obtained, improving the processing efficiency of multimodal generative dialogue tasks.

[0107] In one specific embodiment of this application, fine-tuning the speech output of the second generative dialogue task processing model may include the following steps:

[0108] Step 1: Obtain the actual Mel spectrum frame corresponding to the voice response content sample;

[0109] Step 2: Calculate the first norm of the speech response content sample and the real Mel spectrum frame;

[0110] Step 3: Obtain the predicted duration of the corresponding speech unit of the speech response content sample, and obtain the actual duration of the speech unit corresponding to the actual Mel spectrum frame; where the speech unit is a frame or phoneme;

[0111] Step 4: Calculate the squared Euclidean norms of the predicted duration and the actual duration;

[0112] Step 5: Iteratively train the second generative dialogue task processing model based on cross-entropy loss, first-order norm, and squared Euclidean norm.

[0113] For ease of description, the five steps above can be combined for explanation.

[0114] When fine-tuning the speech output of the second generative dialogue task processing model, the real Mel spectrum frame corresponding to the speech response content sample is obtained, the first norm of the speech response content sample and the real Mel spectrum frame is calculated, the predicted duration of the corresponding speech unit of the speech response content sample is obtained, and the actual duration of the speech unit corresponding to the real Mel spectrum frame is obtained, such as the actual duration of the frame or phoneme corresponding to the real Mel spectrum frame. The square Euclidean norm of the predicted duration and the actual duration is calculated. The second generative dialogue task processing model is iteratively trained based on the cross-entropy loss, the first norm, and the square Euclidean norm. During training, minimizing these three terms simultaneously means that the model must output a speech waveform with natural timbre and reasonable rhythm while generating correct text, thereby achieving end-to-end text and speech consistency.

[0115] The formula for calculating the loss of voice output fine-tuning is as follows:

[0116] ;

[0117] in, It is a standard cross-entropy term, used to maximize the model's ability to generate correct text tags given a multimodal context. The probability, denoted as the model-predicted Mel-spectrum frame, and m represents the actual Mel-spectrum frame. By measuring amplitude differences using the first-order norm, the vocoder maintains linear sensitivity to timbre details. The duration of the predicted frame or phoneme is represented by d, and the actual duration of the frame or phoneme is represented by √(sqrt(d)). norm Emphasizing the squared penalty for duration error encourages speech rate and rhythm to more closely match the target reading. Two weights and Adjusting the relative impact of text and audio errors on the total loss, the former ensures the timbre and clarity of the speech, while the latter fine-tunes the rhythm without excessively amplifying duration jitter.

[0118] S304: Receive input text content, visual content, and streaming voice question content.

[0119] By using multimodal representations that fuse input text, vision, and speech, it provides more comprehensive input information than traditional Voice Activity Detection (VAD) modules (including semantic information processed by the language model, whether the user makes eye contact in the image, etc.), enabling the model to determine in real time whether to interrupt or wait for user input, significantly reducing latency and false positive rate.

[0120] S305: When the duration of silence detected by the full-duplex detection head in the target generative dialogue task processing model reaches the preset silence duration threshold, it is determined that the receiving of the streaming voice question content is complete.

[0121] The full-duplex detection head parameters are randomly initialized and trained on real-world scenarios and synthesized audio over extended periods to enable the model to perform full-duplex dialogue. The training objective function is a binary classification cross-entropy loss, where 0 indicates the user is speaking and 1 indicates the user is not speaking (including when they have finished speaking and background noise). The trained model automatically searches for thresholds on the validation set. In practical applications, the model's output score and the duration threshold for speaking / silence states can be combined for judgment.

[0122] A silence duration threshold can be preset based on the training results. When the duration of the silence state detected by the full-duplex detection head in the target generative dialogue task processing model reaches the preset silence duration threshold, it is determined that the reception of the streaming voice question content is complete. By determining whether the reception of the streaming voice question content is complete based on the preset silence duration threshold, the detection efficiency of the reception status of the streaming voice question content is improved.

[0123] S306: The audio encoder in the target generative dialogue task processing model is used to encode the content of the streaming speech question, and the audio adapter is used to convert the encoded audio feature vector into the first word of the preset unified semantic space.

[0124] S307: Use the text encoder in the target generative dialogue task processing model to encode the text content and obtain the second word in the preset unified semantic space.

[0125] S308: The visual content is encoded using the visual encoder in the target generative dialogue task processing model, and the encoded image feature vector is transformed into a third word in a pre-defined unified semantic space using the visual adapter.

[0126] S309: Utilize the target generative dialogue task processing model to predict and generate response content based on the first, second, and third lexical units to complete this multimodal generative dialogue task processing.

[0127] The embodiments of this application can realize mixed modal context memory in large-scale dialogues or multi-turn tasks by using a unified representation method with a pre-set unified semantic space.

[0128] In one specific embodiment of this application, using a target-generative dialogue task processing model to predict and generate response content based on a first word, a second word, and a third word may include the following steps:

[0129] The target-oriented generative dialogue task processing model is used to predict and generate speech and text responses based on the first, second, and third lexical units.

[0130] After obtaining the first, second, and third lexical units in a predefined unified semantic space, a goal-oriented generative dialogue task processing model is used to predict and generate speech and text responses based on these units. By setting up a model structure that supports full-duplex stop detection at the input end, and mixed input of speech, images, video, and text, and supports simultaneous output of speech and text, integrated text and speech generation is achieved directly from the predefined unified semantic space, realizing end-to-end mixed modality generation. By setting up an end-to-end audio decoder to directly convert semantic vectors into natural and fluent speech waveforms, the network calls and serialization overhead of traditional speech synthesis are eliminated. This improves the rationality and relevance of the generated content.

[0131] Corresponding to the above method embodiments, this application also provides a multimodal generative dialogue task processing apparatus. The multimodal generative dialogue task processing apparatus described below and the multimodal generative dialogue task processing method described above can be referred to in correspondence.

[0132] See Figure 5 , Figure 5 This is a structural block diagram of a multimodal generative dialogue task processing device according to an embodiment of this application. The device may include:

[0133] The content receiving module 51 is used to receive input text content, visual content, and streaming voice question content;

[0134] The first word element acquisition module 52 is used to encode the streaming voice question content using the audio encoder in the target generative dialogue task processing model when the full-duplex detection head in the target generative dialogue task processing model detects that the receiving of streaming voice question content has been completed, and to convert the encoded audio feature vector into the first word element of the preset unified semantic space using the audio adapter.

[0135] The second lexical unit acquisition module 53 is used to encode the text content using the text encoder in the target generative dialogue task processing model to obtain the second lexical unit in the preset unified semantic space.

[0136] The third word element acquisition module 54 is used to encode visual content using the visual encoder in the target generative dialogue task processing model, and to convert the encoded image feature vector into a third word element in a preset unified semantic space using a visual adapter.

[0137] The generative dialogue task processing module 55 is used to predict and generate response content based on the first word, the second word, and the third word using the target generative dialogue task processing model, so as to complete this multimodal generative dialogue task processing.

[0138] As can be seen from the above technical solutions, by embedding the full-duplex detection head into the model, the full-duplex stop detection module is integrated with the model. Through the multimodal fusion representation of input text, audio, and images, it provides more comprehensive input information than traditional full-duplex stop detection modules. This allows the model to determine in real-time whether to interrupt or wait for user input during the decoding stage, significantly reducing latency and false positive rates, and improving the speech stop detection effect. By mapping image feature vectors, audio feature vectors, and text feature vectors to a pre-defined unified semantic space, it achieves temporal consistency parsing of long texts, long videos, and real-time speech, realizing end-to-end mixed modal understanding and reducing model training costs. The unified representation method enables mixed modal contextual memory, improving the understanding ability of the target generative dialogue task processing model and increasing the accuracy of response content.

[0139] In one specific embodiment of this application, the device may further include a model training module, which may include:

[0140] The speech alignment training submodule is used to train the initial generative dialogue task processing model on speech alignment in order to obtain the first generative dialogue task processing model that aligns the input speech and the input text.

[0141] The speech understanding fine-tuning submodule is used to fine-tune the speech understanding of the first generative dialogue task processing model to obtain the second generative dialogue task processing model.

[0142] The speech output fine-tuning submodule is used to fine-tune the speech output of the second generative dialogue task processing model to obtain the target generative dialogue task processing model.

[0143] In one specific embodiment of this application, the speech alignment training submodule may include:

[0144] The speech-text sample pair acquisition unit is used to acquire each speech-text sample pair;

[0145] The vector acquisition unit is used to process the speech vector in each speech-text sample pair using the audio encoder and audio adapter in the initial generative dialogue task processing model to obtain the first vector in the preset unified semantic space, and to obtain the second vector of the text vector in each speech-text sample pair in the preset unified semantic space.

[0146] The first similarity calculation unit is used to calculate the first similarity between each first vector and each second vector.

[0147] The second similarity calculation unit is used to calculate the second similarity between each second vector and each first vector.

[0148] A bidirectional contrastive learning loss calculation unit is used to calculate the bidirectional contrastive learning loss based on each first similarity and each second similarity.

[0149] The first iterative training unit is used to iteratively train the initial generative dialogue task processing model based on the bidirectional contrastive learning loss.

[0150] In one specific embodiment of this application, the speech understanding fine-tuning submodule may include:

[0151] The conditional probability acquisition unit is used to input both the prefix and the three-modal representation into the first generative dialogue task processing model during prediction for each position in a preset unified semantic space in a response content sequence of preset length, and obtain the conditional probability of the prediction result of the position.

[0152] The prediction error loss calculation unit is used to calculate the prediction error loss based on the conditional probability corresponding to each location.

[0153] The second iterative training unit is used to iteratively train the first generative dialogue task processing model based on the prediction error loss.

[0154] In one specific embodiment of this application, the generative dialogue task processing module is specifically a module that uses a target generative dialogue task processing model to predict and generate speech response content and text response content based on the first word, the second word, and the third word.

[0155] In one specific embodiment of this application, the voice output fine-tuning submodule may include:

[0156] The cross-entropy loss generation unit is used to obtain the cross-entropy loss of the correct prediction results generated at each position by the second generative dialogue task processing model given a multimodal context.

[0157] The real Mel spectrum frame acquisition unit is used to acquire the real Mel spectrum frame corresponding to the speech response content sample;

[0158] The first-order norm calculation unit is used to calculate the first-order norm of the speech response content sample and the real Mel spectrum frame;

[0159] The duration acquisition unit is used to acquire the predicted duration of the corresponding speech unit of the speech response content sample, and to acquire the actual duration of the speech unit corresponding to the actual Mel spectrum frame; wherein, the speech unit is a frame or a phoneme;

[0160] Square Euclidean Norm Calculation Unit, used to calculate the square Euclidean norm of the predicted duration and the actual duration;

[0161] The third iterative training unit is used to iteratively train the second generative dialogue task processing model based on cross-entropy loss, first-order norm, and squared Euclidean norm.

[0162] In one specific embodiment of this application, a voice question content reception completion detection module is also included. Specifically, the voice question content reception completion detection module is a module that determines that the streaming voice question content reception is complete when the duration of the silence state detected by the full-duplex detection head in the target generative dialogue task processing model reaches a preset silence duration threshold.

[0163] For the method embodiments described above, see [link to relevant documentation]. Figure 6 , Figure 6 This is a schematic diagram of the multimodal generative dialogue task processing device provided in this application. The device may include:

[0164] Memory 332 is used to store computer programs;

[0165] The processor 322 is used to implement the steps of the multimodal generative dialogue task processing method of the above method embodiments when executing a computer program.

[0166] For details, please refer to Figure 7 , Figure 7 This is a schematic diagram illustrating the specific structure of a multimodal generative dialogue task processing device provided in this embodiment. This device can vary significantly depending on its configuration or performance. It may include a processor (central processing unit, CPU) 322 (e.g., one or more processors) and a memory 332. The memory 332 stores one or more computer programs 342 or data 344. The memory 332 can be temporary or persistent storage. The program stored in the memory 332 may include one or more modules (not shown in the diagram), each module including a series of instruction operations on the data processing device. Furthermore, the processor 322 may be configured to communicate with the memory 332 and execute the series of instruction operations stored in the memory 332 on the multimodal generative dialogue task processing device 301.

[0167] The multimodal generative dialogue task processing device 301 may also include one or more power supplies 326, one or more wired or wireless network interfaces 350, one or more input / output interfaces 358, and / or one or more operating systems 341.

[0168] The steps in the multimodal generative dialogue task processing method described above can be implemented by the structure of the multimodal generative dialogue task processing device.

[0169] Corresponding to the above method embodiments, this application also provides a computer-readable storage medium storing a computer program, which, when executed by a processor, can perform the following steps:

[0170] The system receives input text content, visual content, and streaming voice questions. When the full-duplex detection head in the target generative dialogue task processing model detects that the streaming voice questions have been received, the audio encoder in the target generative dialogue task processing model encodes the streaming voice questions and uses an audio adapter to convert the encoded audio feature vector into a first lexical unit in a preset unified semantic space. The text encoder in the target generative dialogue task processing model encodes the text content to obtain a second lexical unit in the preset unified semantic space. The visual encoder in the target generative dialogue task processing model encodes the visual content and uses a visual adapter to convert the encoded image feature vector into a third lexical unit in the preset unified semantic space. The target generative dialogue task processing model predicts and generates response content based on the first, second, and third lexical units to complete this multimodal generative dialogue task processing.

[0171] The computer-readable storage medium may include various media capable of storing program code, such as USB flash drives, portable hard drives, read-only memory (ROM), random access memory (RAM), magnetic disks, or optical disks.

[0172] For a description of the computer-readable storage medium provided in this application, please refer to the above method embodiments; further details will not be repeated here.

[0173] The various embodiments in this specification are described in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably. For the apparatuses, devices, and computer-readable storage media disclosed in the embodiments, since they correspond to the methods disclosed in the embodiments, the descriptions are relatively simple; relevant parts can be referred to the method section.

[0174] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the technical solutions and core ideas of this application. It should be noted that for those skilled in the art, several improvements and modifications can be made to this application without departing from the principles of this application, and these improvements and modifications also fall within the protection scope of this application.

Claims

1. A multi-modal generative dialog task processing method, characterized in that, The method comprises the following steps: receiving input text content, visual content and streaming voice question content; when the full-duplex detection head in the target generative dialogue task processing model detects that the streaming voice question content receiving is completed, encoding the streaming voice question content by using the audio encoder in the target generative dialogue task processing model, and converting the encoded audio feature vector into the first word piece in the preset unified semantic space by using the audio adapter; encoding the text content by using the text encoder in the target generative dialogue task processing model to obtain the second word piece in the preset unified semantic space; encoding the visual content by using the visual encoder in the target generative dialogue task processing model, and converting the encoded image feature vector into the third word piece in the preset unified semantic space by using the visual adapter; predicting and generating response content according to the first word piece, the second word piece and the third word piece by using the target generative dialogue task processing model to complete the current multi-modal generative dialogue task processing; The method further comprises a training process of the target generative dialogue task processing model, which comprises the following steps: performing speech alignment training on an initial generative dialogue task processing model to obtain a first generative dialogue task processing model in which input speech and input text are aligned; performing speech understanding fine-tuning on the first generative dialogue task processing model to obtain a second generative dialogue task processing model; performing speech output fine-tuning on the second generative dialogue task processing model to obtain the target generative dialogue task processing model; wherein predicting and generating response content according to the first word piece, the second word piece and the third word piece by using the target generative dialogue task processing model comprises the following steps: predicting and generating voice response content and text response content according to the first word piece, the second word piece and the third word piece by using the target generative dialogue task processing model; performing speech output fine-tuning on the second generative dialogue task processing model comprises the following steps: obtaining the cross-entropy loss of the correct prediction result of each position generated by the second generative dialogue task processing model under a given multi-modal context; obtaining the real mel-spectrum frame corresponding to the voice response content sample; calculating the first norm of the voice response content sample and the real mel-spectrum frame; obtaining the predicted duration of the corresponding speech unit of the voice response content sample, and obtaining the real duration of the corresponding speech unit of the real mel-spectrum frame; wherein the speech unit is a frame or a phoneme; calculating the squared Euclidean norm of the predicted duration and the real duration; iteratively training the second generative dialogue task processing model according to the cross-entropy loss, the first norm and the squared Euclidean norm. 2.The multi-modal generative dialog task processing method of claim 1, wherein, performing speech alignment training on an initial generative dialogue task processing model comprises the following steps: obtaining a pair of speech and text samples; For each speech vector in each speech-text sample pair, the audio encoder and the audio adapter in the initial generative dialogue task processing model are used for processing to obtain a first vector in the preset unified semantic space, and a second vector of a text vector in each speech-text sample pair in the preset unified semantic space is obtained; For each first vector, a first similarity between the first vector and each second vector is calculated respectively; For each second vector, a second similarity between the second vector and each first vector is calculated respectively; A bidirectional contrastive learning loss is calculated according to the first similarities and the second similarities; The initial generative dialogue task processing model is iteratively trained according to the bidirectional contrastive learning loss. 3.The multi-modal generative dialog task processing method of claim 1, wherein, The first generative dialogue task processing model is fine-tuned for speech understanding, including: For each position of the preset unified semantic space in a response content sequence of a preset length, a prefix and a three-modal representation are input into the first generative dialogue task processing model during prediction to obtain a conditional probability of a prediction result of the position; A prediction error loss is calculated according to the conditional probabilities corresponding to each position respectively; The first generative dialogue task processing model is iteratively trained according to the prediction error loss.

4. The multi-modal generative dialog task processing method according to any one of claims 1 to 3, characterized in that, Further comprising a detection process of completion of receiving the streaming voice question content, the detection process of completion of receiving the streaming voice question content comprising: When a silence state detected by a full-duplex detection head in the target generative dialogue task processing model lasts for a preset silence duration threshold, it is determined that the receiving of the streaming voice question content is completed.

5. A multi-modal generative dialog task processing apparatus, characterized by, Comprising: A content receiving module configured to receive input text content, visual content and streaming voice question content; A first token obtaining module configured to, when the receiving of the streaming voice question content is detected to be completed by a full-duplex detection head in the target generative dialogue task processing model, encode the streaming voice question content by using an audio encoder in the target generative dialogue task processing model, and convert an audio feature vector obtained by the encoding into a first token in a preset unified semantic space by using an audio adapter; A second token obtaining module configured to encode the text content by using a text encoder in the target generative dialogue task processing model to obtain a second token in the preset unified semantic space; A third token obtaining module configured to encode the visual content by using a visual encoder in the target generative dialogue task processing model, and convert an image feature vector obtained by the encoding into a third token in the preset unified semantic space by using a visual adapter; A generative dialogue task processing module configured to predict and generate response content according to the first token, the second token and the third token by using the target generative dialogue task processing model to complete the current multi-modal generative dialogue task processing; Further comprising a model training module, the model training module comprising: A speech alignment training sub-module configured to perform speech alignment training on an initial generative dialogue task processing model to obtain a first generative dialogue task processing model in which input speech and input text are aligned. The voice understanding fine-tuning submodule is configured to fine-tune the first generative dialogue task processing model to obtain a second generative dialogue task processing model. The voice output fine-tuning submodule is configured to fine-tune the second generative dialogue task processing model to obtain a target generative dialogue task processing model. The generative dialogue task processing module is specifically configured to predict and generate voice response content and text response content according to the first wordpiece, the second wordpiece and the third wordpiece by using the target generative dialogue task processing model. The voice output fine-tuning submodule includes: The cross-entropy loss generation unit is configured to obtain cross-entropy loss of correct prediction results of each position generated by the second generative dialogue task processing model when a given multi-modal context is given. The real mel-spectrum frame acquisition unit is configured to obtain real mel-spectrum frames corresponding to the voice response content sample. The first-order norm calculation unit is configured to calculate a first-order norm of the voice response content sample and the real mel-spectrum frames. The time length acquisition unit is configured to obtain a predicted duration of a corresponding voice unit of the voice response content sample and a real duration of a corresponding voice unit of the real mel-spectrum frames, wherein the voice unit is a frame or a phoneme. The squared Euclidean norm calculation unit is configured to calculate a squared Euclidean norm of the predicted duration and the real duration. The third iterative training unit is configured to iteratively train the second generative dialogue task processing model according to the cross-entropy loss, the first-order norm and the squared Euclidean norm.

6. A multi-modal generative dialog task processing device, characterized by, The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-modal generative dialogue task processing method according to any one of claims 1 to 4. The computer readable storage medium stores a computer program, and the computer program is executed by the processor to implement the steps of the multi-modal generative dialogue task processing method according to any one of claims 1 to 4. ​ 7. A computer-readable storage medium, characterized in that, ​

Images