Multi-mode intelligent question answering and recommending system supporting emotional speech output

Abstract

The invention provides a multi-modal intelligent question-answering and recommendation system supporting emotional speech output, and relates to the technical field of visual question-answering, and the system specifically comprises a multi-modal question-answering module, a multi-modal recommendation module, a speech recognition module, and a speech synthesis module. Wherein the multi-mode question answering module converts an image input by a user into a natural language form, and final question answering is completed in combination with a comet knowledge base COMET; the voice recognition module is used for recognizing voice data input into the multi-mode intelligent question answering and recommending system and processing the voice data into a text form; the multi-modal recommendation module identifies whether modal missing exists in input data or not, generates representation of the missing modal, obtains vector representation of the user and the article through a graph convolutional neural network and learning according to an interaction matrix of the user and the article, carries out reliability calculation, and finally sends a multi-modal recommendation result to the user and the article. Sending back to the multi-mode question and answer module; and the voice synthesis module outputs the voice to the user.

Description

Technical Field

[0001] The present invention relates to the field of visual question answering technology, and in particular to a multimodal intelligent question answering and recommendation system supporting emotional speech output. Background Art

[0002] Visual question answering has attracted increasing attention in the past few years. The latest research in traditional visual question answering can be roughly divided into the following categories: better visual features, more powerful model architectures, and more effective learning paradigms. Most of the current state-of-the-art visual question answering methods adopt the Transformer architecture. By combining visual-language pre-training on large-scale datasets, they have approached or even surpassed human-level performance on several representative benchmark datasets. Some recent studies have considered how to incorporate knowledge into visual question answering. These authors studied how to incorporate knowledge bases into VQA datasets, while providing a set of relevant knowledge for each question. A novel dataset, OK-VQA, requires external knowledge for all questions. This dataset can be used to evaluate the reasoning ability of models in scenarios where answers cannot be obtained by simply looking at the image. Retrieval-based visual question answering The core of this task lies in the acquisition and integration of knowledge. It is natural to think of adding a separate retrieval module to call the required explicit knowledge as external input. Some studies use off-the-shelf visual detection models to generate image labels for knowledge retrieval, while other studies train retrieval models through supervised methods or auxiliary tasks such as entity linking. The Internet is considered to be the simplest source of external knowledge. Using search engines to retrieve relevant corpora as weak knowledge samples is a common method, but this method is limited by network latency, and unstructured knowledge may lead to a decrease in knowledge density, which ultimately reduces the effect. Therefore, some other studies focus on obtaining explicit knowledge from multiple open domain knowledge resources, such as ConceptNet and Wikipedia, and integrating the retrieved knowledge with image question pairs for answer prediction. Considering that high-density knowledge is stored in structured knowledge graph triples, Medhini et al. constructed context-aware subgraphs from large-scale knowledge graphs based on entity name matching or embedding similarity. But they all retain the original graph structure, which is considered to be unable to fully utilize all its information to a certain extent.

[0003] In the past two years, inspired by the great potential of large language models (such as GPT-3) in knowledge reasoning, some recent advanced methods such as PICA use large language models as implicit knowledge sources. Different from the previous pre-training and fine-tuning paradigm, large language models such as GPT-3 introduce a novel in-context learning paradigm. Using prompts, large language models can be adapted to different tasks without modifying any parameters, and they show strong zero-shot and few-shot capabilities. This attractive feature has made large language models successful in many natural language processing tasks, such as classification tasks and dialogue systems. Experiments show that effective small-shot learning requires a huge model, especially the recent visual language model that combines large language models with visual encoders has shown amazing success in various multimodal tasks, including but not limited to Flamingo and BLIP-2. Due to its huge number of parameters and training costs, it is impossible for most people to reproduce its results. Therefore, for methods using contextual learning, building a reasonable and effective prompt is the focus of current research.

[0004] In addition to the above functions, visual question answering can also be used as a multimodal recommendation system. This question-answering recommendation system can directly understand the user's needs and attitudes towards certain attributes during the conversation, so as to make appropriate recommendations. In particular, the emergence of GPT has enabled question-answering recommendation systems to flourish. The more commonly used technical frameworks of conversational recommendation systems (CRS) include natural language understanding modules, dialogue manager modules, recommendation engine modules, natural language generation modules, and user feedback modules. Its practical applications include: Alibaba's "Tao Xiaomi" uses BERT to parse users' natural language input, such as "What clothes are suitable for spring?" to extract intent and keywords from it. Use DIN (Deep Interest Network) to model user historical purchase behavior and recommend similar products based on the user's current input. Combine dialogue state tracking (DST) to achieve multi-round interaction. For example, when a user asks "Are there any other colors?", the system can dynamically adjust the recommendation results according to the context. Netflix uses BERT4Rec and Wide&Deep models to serialize user viewing behavior and predict preferences based on users' historical viewing records. Support complex multi-round dialogues through generative language models (such as GPT-3). Users can ask questions, such as "What cartoons are suitable for children?" The system can explain the reasons for the recommendation. Feature extraction is performed on visual content such as movie cover images and trailers for content matching and recommendation.

[0005] Currently, many visual question-answering systems also provide speech recognition functions. ASR (automatic speech recognition) generates corresponding transcribed texts from given speech. In recent years, with the development of deep learning, speech recognition has gradually transitioned from the previous HMM-GMM-based technical solutions to end-to-end neural network technical solutions. CTC combined with neural encoders solves the problem of different lengths of speech and its corresponding transcribed texts. It does not require time alignment between the input sequence and the output label, but calculates the sum of the probabilities of all possible paths to it through a forward-backward dynamic programming algorithm. This is very important for tasks such as speech recognition, because it is difficult to obtain accurate time alignment annotations in practical applications. RNN-T introduces the idea of ​​transcribers, allowing the model to automatically learn the alignment relationship between input and output during training without explicit time alignment annotations. This greatly simplifies the data preparation process and enables the model to better handle unaligned data. Since RNN-T uses autoregressive reasoning, it naturally supports streaming recognition.

[0006] Text to speech (TTS) is to generate speech waveforms from given text. Usually, excluding text regularization and data annotation, TTS systems usually consist of two steps. First, the acoustic model is used to convert text features into audio features, and then the vocoder is used to convert audio features into audio waveforms. This requires training the models of the two stages separately and then connecting them in series during reasoning. In order to improve the fluency and expressiveness of speech generation results, conditional information of emotions or speaker characteristics can be added to the speech generation process to guide the training of the generation model. Since most data sets currently use discrete emotion labels, in order to achieve smoother and more realistic emotional speech synthesis functions, it is necessary to model emotional features in a high-dimensional or continuous space. Yuxuan et al. trained the global style vector (gst) using attention on the structure of Tacotron and embedded it into the generation model to model different prosodic features. These embeddings are trained without explicit labels and learn how to model a wide range of acoustic features during training. The soft interpretable labels generated by gst can be used to control speech synthesis and style switching in novel ways. When trained on noisy, unlabeled discovery data, gst learns to decompose noise and speaker identity, providing a path for highly scalable but robust speech synthesis. Chae-Bin et al. fused emotional features by adding an additional emotional rendering module to the structure of fastspeech2. It synthesizes expressive emotional speech through phoneme emotional intensity information. The intensity of emotion is calculated by an intensity quantizer based on distance (linear discriminant analysis). The function of manually adjusting the label to control the intensity of emotion is realized by using quantitative labels of different intensities for different emotions.

[0007] A large-scale pre-trained language model is used as a source of knowledge, but the model performance is affected by the model size and the quality of the prompt template. The order and design of the prompt samples are highly sensitive to the generation effect. The image needs to be converted into natural language form, and designing a high-quality prompt template is a key issue. There is still room for optimization of the prompts in the current system. The current image description model mostly generates general descriptions, which may ignore the image details that the question-answering task focuses on. Therefore, it is necessary to train a question-oriented image description model to generate more detailed content to help the large language model better answer questions. The pre-trained language model has insufficient knowledge reserves on specific categories of questions, and many current systems do not make good use of external knowledge to make up for this.

[0008] When using the visual question-answering system to implement multimodal recommendations, there are limitations on the types of multimodal recommendations. Since many models generate content based on the user's historical behavior, the system does not actively explore the user's potential interests. Due to the uneven distribution of content in the system, popular content is often more likely to be recommended, and unpopular content is not exposed for a long time. The needs of niche users cannot be met, reducing the coverage of the system. Long-tail content cannot be fully utilized, limiting the diversity of recommendations, causing users to lose interest in the platform, and ultimately causing customer loss.

[0009] Regarding the input data for multimodal recommendation, due to the low quality of the data uploaded by users during the conversation, such as blurred images, occlusions, too much noise, etc., the input voice data may be unclear or interfered by background sounds, or there may be no data for a certain modality. These modal defects or modality missing situations will affect the model's final alignment of each modality, resulting in the model being unable to make accurate recommendations. On the other hand, in the case of such modality missing, the system cannot collect effective information, resulting in recommendations deviating from user needs.

[0010] In visual question answering, speech input is performed. At this time, in terms of speech recognition, CTC assumes that each symbol in the output sequence is independent, which ignores the dependencies between symbols. Therefore, CTC is relatively weak in processing language models and grammatical structures, which may lead to less fluent recognition results, especially in scenarios that require stronger language model support. In addition, the CTC model usually requires the entire input sequence for decoding and does not support streaming recognition. The training and reasoning process of the RNN-T model is relatively complex, especially when processing long sequences. The joint network needs to process the outputs of the encoder and predictor, which may result in higher computational costs. This makes it difficult to deploy RNN-T on resource-constrained devices, especially on mobile devices or embedded systems.

[0011] In speech synthesis, the solution proposed by Yuxuan et al. learns vectors representing different styles during training, and obtains the heaviest emotional embedding by assigning different weights during inference. This method can more accurately grasp the emotional information in audio features. However, these vectors have poor interpretability, making it difficult to manually select the emotions that users want, and because the autoregressive tacotron2 is used as the base model, it is not effective in terms of generation speed and long text generation. The model (EMOQ-TTS) proposed by Chae-Bin et al. uses linear discriminant analysis to calculate the relative distance between different emotions during training, and quantizes the distance between each emotion and neutral emotion. Although an infinite number of quantization results can be set to model emotions at any intensity, the intensity expression of emotions in reality is not linearly enhanced and weakened, and the effect of the generated model is slightly reduced for unseen emotional quantization embeddings.

[0012] When building a visual question answering system, it is necessary to use a large-scale pre-trained language model as a source of knowledge, but the model performance is affected by the model size and the quality of the prompt template. The order and design of the prompt samples are highly sensitive to the generation effect. The image needs to be converted into a natural language form, and designing a high-quality prompt template is a key issue.

[0013] The current image description models mostly generate general descriptions, which may ignore the image details that the question-answering task focuses on. Therefore, it is necessary to train a question-oriented image description model to generate more detailed content to help the large language model better answer questions.

[0014] The pre-trained language model has insufficient knowledge reserves for specific categories of problems, and it is necessary to select a suitable external knowledge base to supplement it in order to improve model performance.

[0015] When using the question-answering system for recommendation, choose to use graph convolution to implement the recommendation. In the process of using the graph convolutional neural network algorithm for multimodal recommendation, because the core feature of the graph convolutional neural network is to extract local features through convolution operations, such as local patterns of user behavior or specific attributes of items, this only captures similarities, resulting in the recommendation content being concentrated in a range that is very similar to the user's historical interests and lacking diversity. Secondly, in the process of multimodal recommendation, if the content of a certain modality input is incomplete or a certain modality is not directly input, then this will affect the subsequent recommendation process. Most conventional multimodal recommendation systems do not consider the occurrence of this situation.

[0016] For the speech recognition module, the accuracy of speech recognition is improved and the model complexity is reduced. Since the CTC model has relatively weak modeling capabilities in language model and grammatical structure and does not support streaming recognition, and the model based on RNN-T as the main idea usually has a slow recognition speed due to the autoregressive seq2seq operation.

[0017] For the speech synthesis module, the existing solutions have room for improvement in the rhythm, emotional expression and controllability of the generated speech. Most models rely on discrete labels for modeling style and emotion. Although this method can accurately obtain the target speech, it cannot give users more humane responses in continuous emotional expression, which feels rather stiff. In addition, due to the previous model's modeling problems with the spectrogram, there is still room for improvement in the rhythm and style modeling effects. Summary of the invention

[0018] In view of the deficiencies of the prior art, the present invention provides a multimodal intelligent question-answering and recommendation system that supports emotional speech output.

[0019] A multimodal intelligent question-answering and recommendation system supporting emotional speech output specifically includes a multimodal question-answering module, a multimodal recommendation module, a speech recognition module, and a speech synthesis module.

[0020] The multimodal question answering module includes: a question-oriented image description generation module, an external knowledge context expansion module and a question-aware visual question answering module; The question-oriented image description generation module is used to convert the image input by the user into a natural language form; the module uses the BLIP2 model to receive the image, the question and the instruction with the user input, locates the corresponding object from the image according to the input question, and outputs the natural language of the image description oriented to the user input; The external knowledge context extension module provides common sense knowledge, and uses the Comet knowledge base COMET to provide common sense knowledge; the Comet knowledge base COMET contains entities and relationships between entities, and focuses on social common sense centered on events, including the motivations and influences of event participants; the motivation refers to the internal driving force that determines the behavior of the participants; the influence refers to the changes that the behavior of the participants will cause to other events and entities; The question-aware visual question-answering module is used to complete the final question answering. The module uses the contextual learning ability of Zhipu Qingyan AI large model ChatGLM for visual question answering, and specifically includes two components: an answer reasoning generation component and a heuristic enhancement prompt component; the answer generation component generates two types of answer reasoning, namely, answer candidate sets and answer-aware examples; the heuristic enhancement prompt component integrates heuristic answers, questions, and question-oriented image descriptions into formatted prompts to instruct Zhipu Qingyan AI large model ChatGLM to predict answers; The answer candidate set includes several groups of answers, and for each answer perception example, a group of answers is generated using the reasoning ability of ChatGLM; The answer perception example is: according to the image and question combination input by the user, the most similar image-question pair is calculated from the VQA dataset using cosine similarity as the answer perception example; Heuristic answers include answer-aware examples, answer candidate sets, and task instructions.

[0021] The multimodal question-answering module specifically performs the following steps: Step A1: The question-oriented image description generation module receives images and questions input by the user and outputs image descriptions, treating the existing VQA2.0 dataset as a natural source of tasks and task-related visual detail pairs; synthesizing question-aware captions by combining image captions and question-answer pairs using the contextual learning of the Zhipu Qingyan AI model ChatGLM; the prompts of the Zhipu Qingyan AI model ChatGLM contain task instructions, human-written examples, and VQA question-image pairs for synthesizing task-aware captions; the examples include questions, image descriptions, and answers; Step A1.1: Generate an image description training set using ChatGLM, where the image description dataset contains images and descriptions; each image is represented by concatenating 5 human-written COCO captions; Step A1.2: Extract 5 candidate subtitles for each question-answer pair from ChatGLM; design a pipeline to filter out the best candidate subtitles as training samples for the question-oriented image description generation module; for each candidate subtitle, use the soft VQA accuracy to compare the answer corresponding to the candidate subtitle with the true answer to soft-score the candidate subtitle; the soft VQA accuracy is obtained by the character error rate, and the candidate subtitles are sorted according to the soft score, and the candidate subtitle with the highest soft score is selected to generate the training sample; The soft accuracy is obtained by the following formula: ; where g i refers to groundtruth, which is the standard answer, a refers to answer, which is the answer given by the system, n is the number of standard answer sets for each question, x, y, z refer to the results calculated based on CER, and the current result is the data in the answer set. The lowest three are taken and identified as x, y, z, and CER is the character error rate; Step A1.3: For the question-oriented image description generation module, the training samples obtained in step A1.2 are reassembled into instruction prompts in the form of question-title pairs using the BLIP2 model; Step A2: Introduce external common sense knowledge for the visual question answering task through the external knowledge context extension module, use the knowledge base generation model COMET to generate contextualized common sense knowledge related to the question and image content, and filter and extract the required reasoning; Step A2.1: Generate common sense reasoning using the COMET model. First, the COMET model performs language processing on the question and converts it into a declarative sentence. Then, the object detection model YOLO is used to extract object labels from the image and combine the object labels with the question to form a question object phrase. Step A2.1.1: Use the COMET model to generate common sense reasoning based on the input phrase; Step A2.1.2: Clean up repeated or redundant inferences by detecting vocabulary overlap and removing sentences with similarity greater than a set threshold; Step A2.2: After generating the inferences, select the most useful information from them. Specifically, use SBERT to encode them in the same vector space, sort them by cosine similarity, and select only the top 5 most relevant inferences by calculating the problem and each inference; Step A3: The question-aware visual question answering module uses a portion of the VQA examples as task examples for ChatGLM and concatenates the contextual learning examples together to form a prompt; each contextual learning example contains a question, a generated image description, and an answer; the user-input question is appended to the contextual learning example and input to ChatGLM; ChatGLM generates a predicted answer based on an open text generation method.

[0022] The speech recognition module is used to recognize the speech data input into the multimodal intelligent question-answering and recommendation system and process it into text form; specifically, a fully end-to-end speech recognition model structure based on Transformer is used.

[0023] The speech recognition module specifically performs the following steps: Step B1: preprocessing the original audio data; Extract the audio feature FBank of the original audio. At the same time, preprocess the transcribed text corresponding to the original audio, convert the text content into discrete numbers, form a character index sequence, form the dictionary index corresponding to each character, and obtain the feature vector of the character index sequence through embedding; Step B2: Process the audio feature Fbank using a one-dimensional resnet structure, wherein the one-dimensional resnet structure includes 12 one-dimensional dilated causal convolutions with a convolution kernel size of 3 using residual links, to obtain audio features for speech recognition as input to the encoder; Step B3: Input the audio features for speech recognition into the encoder, further abstract the input features, and use 6 self-attention layers to learn context information to obtain intermediate features for decoding; Step B4: Use the transformer decoder structure and 6 cross-attention layers to train the intermediate features and encoder input. Map the output tensor to a vector of the same dimension as the character dictionary through a fully connected layer. Get the character in the dictionary corresponding to the index of the maximum value at each time step. Specifically, add the starting character before the input character index sequence. <sos>, and add the end character at the end of the character index sequence <eos>, and add masks in the calculation process of cross attention, using the structure of a fully end-to-end speech recognition model structure based on Transformer to achieve speech recognition end-to-end.

[0024] The multimodal recommendation module includes a graph convolution module, a data augmentation module and a cross-modal translation module; The cross-modal translation module identifies whether there is a modality missing in the data passed into the multimodal recommendation module. If there is a modality missing, the cross-modal translation module completes the translation task; the modality missing refers to the lack of one or two of the three modalities of text, audio, and vision. The cross-modal translation module will learn the commonalities and characteristics of the same item in different modalities based on the representation of the passed-in modality by using the multi-layer perceptron model MLP, thereby using the existing modality to generate a representation of the missing modality, which is then passed into the graph convolution module.

[0025] The graph convolution module learns the vector representation of users and items through the graph convolutional neural network according to the interaction matrix of users and items; performs the same graph convolution operation on different modalities, and finally obtains the embedded representation of users and items in the three modalities of text, vision, and speech respectively; fuses the embedded representation through the multi-head attention algorithm to obtain the embedded representation of users and items after modal fusion and .

[0026] The data augmentation module performs reliability calculation on the original user-item interaction matrix; the so-called reliability calculation is based on the user-item interaction matrix G , add weights to the edges in the user-item interaction graph to obtain the calculated user-item interaction matrix G’ ; The user-item interaction matrix G In the matrix, if a user interacts with an item, the corresponding position in the matrix is ​​1, otherwise it is 0; the weight is calculated by calculating the similarity between items based on all items that each user has interacted with; according to the calculated user and item interaction matrix G’ , create a data augmentation graph ; The data augmentation graph is created Specifically, among users and items that have not interacted, a random edge is established for them, and the weight of this edge is the calculated reliability; the data augmentation graph is obtained , and then augment the graph for the data Perform graph convolution and multi-head attention operations to obtain new embedding representations of users and items, recorded as and ; At this time, by designing contrastive learning, the embedding representation of users and items obtained in the graph convolution module is and , learning from data augmented graphs Convolution and The above diverse information finally presents the result of multimodal recommendation, namely and After being sent into the module, it is sent back to the multimodal question and answer module.

[0027] The multimodal recommendation module specifically performs the following steps: Step C1: Get the embedding representation of items of different modalities , randomly generated user embedding representation The interaction matrix between users and items is sent to the multimodal recommendation module. , representing the three modes of text, vision, and speech respectively; in the multimodal recommendation module, the data first enters the cross-modal translation module; the operations to be performed by the cross-modal translation module are as follows: Step C1.1: Modal feature decomposition; the cross-modal translation module embeds the input , decomposed into two parts: modal shared features and modality-specific features ; The modalities share features Used to capture commonalities between different modalities; Modality specific features Used to express the unique characteristics of a modality; Step C1.1.1: Modality shared feature learning: learn the modality shared features between different modalities through supervised contrastive learning loss function; The contrastive learning loss function is as follows: ;in, P ( i ) represents all items that belong to the same category or have similar characteristics as item i, A ( i ) represents all items in the current batch, a∈ A ( i ) represents the items in the current batch except item i itself; the representation of items is divided into modal shared features and modality-specific features Two parts, then use the projection head Will and After concatenation, it is projected into the embedding space shared by multiple modalities, z i 、z p 、z p Both refer to the use of projection head The embedding representation of the specified item and The content obtained by projecting into the shared embedding space, p represents the subscript of another item in the same category as i, and τ is the scalar temperature parameter; Step C1.1.2: Modality-specific feature learning: learning is performed by maximizing the dissimilarity between modalities and modality-shared features, using negative L2 distance as the L2 loss function; The L2 loss function is specifically: ;in, M is the number of different modes, and It means the i modality-specific and shared features of the modalities; Step C1.1.3: If adding the distance between the modality-shared features and the modality-specific features does not produce the best results, design a loss function to narrow the distance between the transformed embedding representation and the true embedding representation through a multi-layer perceptron; The loss function is specifically: i The following features are obtained through a multi-layer perceptron j The embedding representation under the modality is as follows: ; Where: MLP is a multi-layer perceptron, E i and E j Representing embedded representations under different modalities; Step C1.2: Train the total loss function and divide the different modal embedding representations of the items into modality-shared features and modality-specific features by executing step C1.1; The total loss function is as follows: ; where α con , α dis , α trans is the learnable coefficient; When a modality is missing, it is sent to the cross-modal translation module to obtain the prediction and replacement of the missing modality, and the representation of the missing modality is obtained as shown below: ; Step C2: Add the obtained embedding representations of all modalities of the item, i.e., modality-shared features and modality-specific features, to the user embedding representation , and are sent to the graph convolution module together; for different item modality embedding representations, the same graph convolution operation is performed on the original user-item interaction matrix G; the graph convolution operation uses the lightGCN algorithm, and the specific formula is as follows, recording the embedding representation obtained by each layer of convolution, and then averaging the k representations as the embedding representation of users and items at the current stage: ;in, Represents the embedding vector of node i in the kth layer of the user-item interaction matrix G. The embedding representation in the initial state is the embedding representation of items of different modalities obtained in the cross-modal translation module. For the embedding representation of items of different modalities, when performing graph convolution operations, the initial embedding representation of the user is randomly generated based on the user ID. The table represents the number of neighbors of node i, that is, the degree of the node on G; Step C3: Input the original user-item interaction matrix into the data augmentation module to perform reliability calculation; specifically, randomly sample the user-item pairs that are not in the original interaction matrix, add them to the original user-item interaction graph, and add reliability weights to generate a new interaction matrix based on the embedding representation of all modalities of the item plus the user embedding representation , in the data augmentation graph Perform graph convolution on the image to obtain a new embedding representation, and then use this new embedding representation to calculate a new loss function; The reliability calculation is specifically as follows: ; The new loss function is as follows: ;in, and Respectively represented in the user-item interaction matrix G And data augmentation graph The user embedding obtained after convolution is represented by U, U represents all users, u and v represent two different users in U, f() represents the cosine similarity function, and τ represents the temperature hyperparameter; Step C4: The user and item embedding representations of different modalities obtained by the graph convolution module are sent to the multi-head attention model to fuse the multimodal information. The scores are calculated after multiplying the embedded representations of the fused user and item, and the item information with the highest score that has not yet interacted with the user is sent to the multimodal question-answering module. The information of items with the highest scores and that have not yet been interacted with by the user is displayed to the user. If text information and images are obtained, they are directly output. If they need to be translated into speech for output, the text data is sent to the speech synthesis module.

[0028] The speech synthesis module includes an emotion VAE sampling module, an emotion fusion module and an end-to-end decoder module.

[0029] The emotion VAE sampling module includes an audio encoder, an emotion encoder and a decoder. The audio encoder is stacked by three layers of convolutional neural networks with residual links, which are used to obtain the mean and variance of the Mel-spectrogram. The Mel-spectrogram is obtained from an audio file. The audio is first sampled at a sampling rate of 22k, and then a short-time amplitude spectrum is obtained through pre-emphasis, framing, windowing and short-time Fourier transform. The short-time amplitude spectrum is passed through a Mel filter group to obtain a Mel-spectrogram. The emotion encoder is composed of a fully connected layer, which is used to obtain the mean and variance information in the emotion feature. The emotion feature is a vector corresponding to a discrete emotion label. The mean and variance of the audio spectrogram feature are calculated by KL divergence with the mean and variance of the emotion feature, and the mean and variance of the audio spectrogram feature are used to sample the emotion feature vector; wherein the KL divergence is used as part of the overall loss calculation of the speech synthesis module; the decoder is composed of two fully connected layers, which are used to linearly transform the sampled emotion features.

[0030] The emotion fusion module includes a diffusion model with a unet network as a Gaussian noise predictor and multiple kernel predictors. The diffusion model with the unet network as a Gaussian noise predictor uses a convolution-based emotion embedding method in the downsampling process, and performs convolution operations on text features and Gaussian noise required by the diffusion model. A kernel predictor is designed in the process of downsampling each layer of the unet network. The kernel predictor first maps the emotion feature dimension into a high-dimensional vector through a fully connected layer, and then transforms the high-dimensional vector into the shape of a pair of convolution kernels required for parameters. A pair of convolution kernels perform convolution operations on text features respectively, and then multiply the results to obtain the output of the fusion of the kernel predictor and the text features of this layer. In the process of three-layer downsampling in the unet network, one-dimensional convolutions with expansion coefficients of 1, 2, and 3 are used respectively. This process is expressed as: , ; Where x represents the text feature, e represents the expanded sentiment feature, and represents a pair of emotion convolution kernels generated simultaneously, z represents the feature after emotion fusion, and σ represents the relu activation function.

[0031] The end-to-end decoder module includes a generator module and a discriminator module; the generator module is used to convert the audio spectrogram into a time domain waveform, and includes four layers of convolutional neural networks with different expansion coefficients and residual links, and records the output results of each layer of the neural network; the discriminator module is used to perform sentiment classification and distinguish audio differences, and specifically has two training objectives, namely, the difference calculation between the synthetic audio and the real audio, and the difference between the sentiment classification label of the generated audio and the real label, wherein the difference calculation between the synthetic audio and the real audio is specifically: for the synthetic audio and the real audio, the generator module is used to calculate the output results of each layer of the neural network and the difference of the real audio after each layer of the neural network processing and the waveform difference with the real time domain audio, and the sentiment classification label of the generated audio is specifically: using the open source timnet model, the audio spectrogram is analyzed through a three-layer bidirectional convolutional neural network, and the difference in the results after the convolution operation of the real audio and the synthetic audio at different expansion coefficients is calculated; the adversarial training loss of the end-to-end decoder module is: , , , ; where x is the generated audio, y is the real audio, T represents the total number of discriminator layers, and D l Represents the feature map of the discriminator layer l, with N features l , k represents the dilation coefficient of the bidirectional causal convolution, is the expected formula after given x and y, is the expectation after given x, G() is the generator module, D() is the discriminator module, n is the number of convolutional layers with different dilation coefficients, is the sentiment classification loss in the discriminator in the decoder module, is the difference loss of each layer of the neural network in the generator module in the decoder module, is the adversarial loss of the generator and the discriminator in the decoder module.

[0032] The speech synthesis module specifically performs the following steps: Step D1: Construct a single-person emotional speech dataset, extract the triples consisting of emotional speech, text, and emotional labels in the single-person emotional speech dataset, regularize the text, remove punctuation and meaningless characters, modify the numbers to the corresponding English expressions, and set the discrete emotional labels to five integers of 0, 1, 2, 3, and 4, representing anger, sadness, neutrality, happiness, and surprise, respectively; Step D2: Convert the text in the triple into a phoneme sequence using the g2p_en interface and obtain its corresponding phoneme feature vector; Step D3: convert the emotional speech in the triple into the corresponding mel spectrogram features; Step D4: Find the phoneme feature vector, obtain the phoneme feature through the text encoder, use the monotone alignment search method to find the path from the one-to-many phoneme feature to the mel spectrum length with the smallest difference, and then expand the phoneme feature sequence according to the duration required for each phoneme feature obtained to make it consistent with the mel spectrogram length; Step D5: Use VAE to implement low-dimensional continuous space sentiment modeling, find the sentiment vector corresponding to the sentiment label in the triple, and obtain the mean and variance of the low-dimensional feature distribution through the prior encoder, where the prior encoder consists of two layers of fully connected layers, uses the reparameterization technique to sample in the low-dimensional latent space, and generates the corresponding sentiment features through the decoder. The decoder is also implemented using two layers of fully connected layers to obtain the sentiment features to guide the generation model; Step D6: stack the phoneme feature sequences with the same length obtained in step D4 and the mel spectrogram after adding Gaussian noise, and then input them into the diffusion probability model as a 2-channel plane tensor. The diffusion probability model uses a Gaussian noise predictor with a unet structure to reconstruct the input. In each downsampling layer of unet, it includes a layer of emotional convolution module, a layer of ordinary convolution module and a layer of downsampling layer, wherein the emotional convolution module includes a layer of emotional convolution and two layers of ordinary convolution, and the ordinary convolution module includes two layers of ordinary convolution. The convolution kernels used in the ordinary convolution operation are all of a size of 3×3. In the emotional convolution, the emotional features are expanded according to the number of input channels × the number of convolution kernel parameters × the number of output channels, and then converted into a 3×3 emotional convolution kernel after two layers of ordinary convolution; The generated emotional convolution kernel is used to perform a convolution operation on the 2-channel plane tensor to obtain the feature tensor to be downsampled; then, the ordinary convolution operation is performed to obtain the downsampled output of this layer; the diffusion probability model takes the phoneme feature sequence and the mel spectrogram with Gaussian noise added as input, and takes the mel spectrogram as output. At the same time, a decoder is trained, which takes the mel spectrogram obtained by real audio processing as input and the time domain audio waveform as output. The decoder uses a hifigan structure, and an emotional speech classification module is added to the discriminator module. The emotional speech classification module uses a bidirectional convolution containing three layers of different expansion coefficients to extract the emotional features of the synthesized audio and the real audio, and guides the generator to express the emotion in the speech by distinguishing the emotional difference between the generated audio and the real audio; Step D7: Finally, the voice is output to the user, where the output includes not only the voice information but also the corresponding text information and pictures.

[0033] The beneficial effects of adopting the above technical solution are:

[0034] The present invention provides a multimodal intelligent question-answering and recommendation system that supports emotional speech output. It generates question-guided image descriptions for question-answering tasks, which are more detailed than general descriptions. It supplements the knowledge of a large language model with the help of a scientific knowledge base and a common sense knowledge base. It designs an answer perception generation module to promote a better understanding of image content. It introduces the common sense knowledge base commonly used in dialogue systems into question-answering as a knowledge supplement. It decomposes the original end-to-end training process into several small parts, so that the effect of an end-to-end model can be achieved even with low resources.

[0035] The present invention aims at the problem of lack of diversity and modality in question-answering multimodal recommendation. To this end, the present invention learns the modality-shared features and modality-specific features between different modalities, realizes mutual translation between different modalities through a multi-layer perceptron, and completes the missing modalities, thereby alleviating the problem of modality loss that may occur in multimodal recommendation, affecting subsequent recommendation results, and failing to fully utilize multimodal information.

[0036] Secondly, existing recommendation models focus on the accuracy of recommendations, ignoring the importance of recommendation diversity. Diversified recommendations can prevent users from falling into information building, fatigue, and customer loss. The present invention uses data augmentation to introduce random and reliable interactions into the original user and item interaction matrix to maximize mutual information, thereby making the final user and item representations diverse.

[0037] In the speech synthesis model, the present invention uses VAE to model emotional features in a low-dimensional continuous space, and achieves the purpose of manually controlling the intensity and range of emotions through sampling, thereby improving the interpretability and fluency of emotional modeling. Combined with the powerful generation capability of the diffusion model, the diffusion model is used to generate the target audio non-autoregressively, and in the process of downsampling, emotional features are integrated at different levels and receptive fields through emotional convolutions with different dilation coefficients, thereby achieving excellent rhythm control. The decoder is trained using adversarial learning, and an emotional analysis module is added to the discriminator, which constrains the emotional expressiveness of the generated speech, simplifies the training steps, and reduces the model inference time and space occupied; BRIEF DESCRIPTION OF THE DRAWINGS

[0038] Figure 1 The overall structure diagram of the multimodal intelligent question answering and recommendation system of the present invention; Figure 2 This is a diagram of the multi-modal recommended module structure of the present invention; Figure 3 The speech synthesis module of the present invention performs a flow chart; Figure 4 This is a flowchart of executing the multimodal question-answering module of the present invention; Figure 5 This is a structural diagram of the diffusion probability model of the present invention. DETAILED DESCRIPTION

[0039] The specific implementation of the present invention is further described in detail below in conjunction with the accompanying drawings and examples. The following examples are used to illustrate the present invention, but are not intended to limit the scope of the present invention.

[0040] A multimodal intelligent question-answering and recommendation system that supports emotional speech output, such as Figure 1 As shown, it specifically includes a multimodal question-answering module, a multimodal recommendation module, a speech recognition module, and a speech synthesis module; The multimodal question answering module includes: a question-oriented image description generation module, an external knowledge context expansion module and a question-aware visual question answering module.

[0041] The question-oriented image description generation module is used to convert the image input by the user into a natural language form; the module uses the BLIP2 model to receive the image, the question and the instruction with the user input, locates the corresponding object from the image according to the input question, and outputs the natural language of the image description oriented to the user input; The external knowledge context extension module provides common sense knowledge, and uses the Comet knowledge base COMET to provide common sense knowledge; the Comet knowledge base COMET contains entities and relationships between entities, and focuses on social common sense centered on events, including the motivations and influences of event participants; the motivation refers to the internal driving force that determines the behavior of the participants; the influence refers to the changes that the behavior of the participants will cause to other events and entities. In this embodiment, the external knowledge context extension module specifically includes two parts: target detection and knowledge context generation. The target detection part receives image input and outputs the current image object expressed in the form of a phrase; the knowledge context generation part receives the current image object expressed in the form of a phrase and a user input question, and outputs the extended knowledge that is most relevant to the current image object and the user input question; The question-aware visual question-answering module is used to complete the final question answering. This module uses the contextual learning ability of Zhipu Qingyan AI large model ChatGLM for visual question answering, and specifically includes two components: an answer reasoning generation component and a heuristic enhancement prompt component; the answer generation component generates two types of answer reasoning, namely answer candidate sets and answer perception examples; the heuristic enhancement prompt component integrates heuristic answers, questions, and question-oriented image descriptions into formatted prompts, instructing Zhipu Qingyan AI large model ChatGLM to predict answers. The input of the question-aware visual question-answering module in this embodiment consists of task instructions, contextual examples, knowledge extensions, and test instances. ChatGLM accepts input and generates answers.

[0042] The answer candidate set includes several groups of answers, and for each answer perception example, a group of answers is generated using the reasoning ability of ChatGLM; In this embodiment, the user input image and question are fed into the original ChatGLM to obtain a preliminary answer, because it is found through experiments that most correct answers appear in the first answer of the model.

[0043] The answer perception example is: according to the image and question combination input by the user, the most similar image-question pair is calculated from the VQA dataset using cosine similarity as the answer perception example.

[0044] The heuristic answer consists of answer perception examples, answer candidate sets and task instructions, which are used to inspire ChatGLM to give more accurate answers. The task instruction refers to the use of ChatGLM's contextual learning ability, which requires writing task instructions to instruct it to complete certain tasks. The task instruction template used in the present invention is: "Please select the correct answer from the candidates based on the heuristic answer, question and image description. The real answer may not be included in the candidates."

[0045] The multimodal question answering module is as follows Figure 4 As shown, perform the following steps: Step A1: The question-oriented image description generation module receives images and questions input by the user and outputs image descriptions, treating the existing VQA2.0 dataset as a natural source of pairs of tasks and task-related visual details; synthesizing question-aware subtitles by combining image captions and question-answer pairs using the Zhipu Qingyan AI large model ChatGLM context learning; the prompt Prompt of the Zhipu Qingyan AI large model ChatGLM contains task instructions, 20 human-written examples, and VQA question image pairs for synthesizing task-aware titles; the examples include questions, image descriptions, and answers.

[0046] In this example, we first write a prompt to guide the large language model to generate candidate captions. The prompt writing follows the following three principles: (1) covering the visual details required for the prompt to be explained, (2) describing the main object like a general caption, and (3) using auxiliary information in the prompt when necessary. For example, assuming that the prompt contains a visual question answering question, it can directly describe the visual content being asked (e.g., for questions about visual details), or provide information that helps downstream models infer the answer (e.g., for questions that require external knowledge to solve). We regard the existing VQA dataset as a natural source of pairs of tasks and task-related visual details. We synthesize question-aware captions by combining general image captions and question-answer pairs using contextual learning capabilities.

[0047] Step A1.1: Generate an image description training set using ChatGLM, which contains images and descriptions. Since ChatGLM only accepts text input, each image is represented by concatenating 5 human-written COCO captions. ChatGLM's common sense reasoning ability allows the model to understand images to a certain extent through COCO captions and synthesize new examples by following human-written examples. The examples are abstracted into the following templates: .

[0048] Step A1.2: To ensure the quality of the generated captions, 5 candidate captions are extracted from ChatGLM for each question-answer pair; a pipeline is specifically designed to filter out the best candidate captions as training samples for the question-oriented image description generation module; the idea is that a pure text question-answering system should correctly answer the question and give a high-quality text-guided caption as context. For each candidate caption, the soft VQA accuracy is used to compare the answer corresponding to the candidate caption with the true answer to softly score the candidate caption; the experiment found that the VQA accuracy error in the open domain generated answer penalizes the answer that is slightly different from the correct answer, for example, when the true answer is "women" and the answer is "woman", the VQA accuracy is 0. To solve this problem, we designed a soft VQA accuracy, which is obtained by the character error rate, sorted the candidate captions according to this soft score, and selected the candidate caption with the highest soft score to generate training samples.

[0049] For fine-tuning tasks, the quality and quantity of the fine-tuning dataset are equally important. In order to ensure the quality of generated subtitles, we designed two screening indicators for generating candidate sets. In this embodiment, a new soft VQA accuracy is designed. Assume that the predicted answer we get is a , the correct answer written manually is , our soft accuracy is given by the following formula: ; where g i It refers to groundtruth, which is the standard answer. n is the number of standard answer sets for each question. x, y, z refer to the results calculated based on CER. The current result is the data in the answer set. The lowest three are taken and identified as x, y, z. CER is the character error rate.

[0050] It is calculated by dividing by the total number of characters in the groundtruth. In contrast, traditional VQA accuracy uses exact matching. We rank the candidate captions based on this soft score. Multiple candidate captions can correctly answer the question and get the same soft score. To break ties, we also calculate the CIDEr score between the candidate captions and the COCO ground truth captions. Among the candidates with the highest soft VQA accuracy, the one with the highest CIDEr score is selected as the final training example.

[0051] Step A1.3: For the question-oriented image description generation module, the BLIP2 model is used to reassemble the training samples obtained in step A1.2 into instruction prompts in the form of question-title pairs. The instruction template is: Using this hint template we fine-tune a problem-oriented BLIP2 image captioning module.

[0052] In this embodiment, the BLIP2 model is trained and the image is converted into image patches that share the same sub-word tag set. Assume that the training sample is , in It is a text prompt. is an image block, is a synthetic task-aware title, the model will as input and trained to generate task-aware captions C i = [ ], c i To describe the captions for n images, we use negative log-likelihood loss and train the model in an end-to-end manner. The training loss is: ; Where logp() is the negative log-likelihood function and p() is the conditional probability; Step A2: Introduce external common sense knowledge for visual question answering tasks through the external knowledge context extension module, especially those that cannot be directly inferred from images. Use the knowledge base generation model COMET to generate contextualized common sense knowledge related to the question and image content, and filter and extract the required reasoning; In this example, we receive an image and a question. Given an image with corresponding image regions I pre-computed using YOLO and a question Q associated with the image, we generate common sense inferences C about events and entities in the question phrase and two object labels O, and select the set of common sense inferences that are most useful for answering the question. = { }; Step A2.1: Generate common sense reasoning using the COMET model, which is a language model specially trained for common sense reasoning tasks, based on large common sense knowledge graphs (such as ConceptNet and ATOMIC). First, the COMET model performs language processing on the question and converts it into a declarative sentence form (such as "What is the purpose of an umbrella?" is converted to "The purpose of an umbrella is"). Then, the object detection model YOLO is used to extract object labels (such as "dog", "chair", etc.) from the image, and the object labels are combined with the question to form a question object phrase (such as "The purpose of an umbrella is, with a dog and a chair").

[0053] Step A2.1.1: Use the COMET model to generate common sense reasoning based on the input phrase; the COMET model generates several reasonings for each relationship type; for example, for "the purpose of an umbrella is", it may generate reasoning such as "umbrellas are used to block rain" and "umbrellas are used to block the sun".

[0054] Step A2.1.2: Since the inferences generated by COMET may contain repeated or redundant content, we clean up repeated or redundant inferences by detecting word overlaps and removing sentences with similarity greater than a set threshold (e.g., inferences with word overlaps exceeding 70%).

[0055] Step A2.2: After generating the inferences, select the most useful information from them to ensure that the model is not disturbed by irrelevant or redundant inferences. Specifically, use SBERT encoding in the same vector space, sort by cosine similarity, and select only the top 5 most relevant inferences by calculating the question and each inference. These inferences are considered to be the most helpful in answering the question.

[0056] This example uses the latest version of COMET, which is initialized using BART in a zero-shot setting. Because COMET is mainly trained on declarative sentences, it is sensitive to declarative sentences. So we first process each question using AllenNLP's constituency parser and convert it into a declarative sentence, such as "What is the purpose of an umbrella?" is changed to "The purpose of an umbrella is". We limit the number of object tags used in the COMET input to two, because adding multiple tags makes the inference more confusing and noisy. In this way, we can obtain inferences that can provide us with additional knowledge about the visual and language inputs.

[0057] Feeding up to 150 COMET inferences into a VL Transformer model is impractical due to the high computational cost and noise associated with feeding such a large amount of text tokens. To rank and select inferences, we employ semantic search based on Sentence Transformers (SBERT), which are pre-trained on the task of retrieving candidate answers to search queries. In this approach, questions and inferences are embedded into the same vector space using SBERT, and the cosine similarity between the question and inference embeddings is used to rank the inferences. The embedding of SBERT is represented as: We label the inference of each question with a similarity score based on the proportion of overlap with the human-annotated answers, and ultimately obtain extended contextual knowledge.

[0058] Step A3: Question-aware Visual Question Answering Module After converting the VQA examples into question-answering examples that ChatGLM can understand in the above steps, use part of the VQA examples as task examples for ChatGLM and concatenate the contextual learning examples together to form a prompt; each contextual learning example contains a question (e.g., "When was this thing first invented?"), a generated image description (e.g., "A train travels along a dirt road"), and an answer (e.g., "1804"). The user-input question is attached to the contextual learning example and fed into ChatGLM; ChatGLM generates a predicted answer based on an open text generation method; The context learning examples are image and text data that are similar to the image and question input by the current user, which are filtered from the local image and text dataset and combined according to the question, image description and answer. In this embodiment, answer candidate subtitles C and external knowledge extension E are used to obtain heuristic enhancement prompts that promote ChatGLM's small amount of learning ability for knowledge-based VQA. A prompt consists of a prompt header, a set of context examples, and a test input. The prompt header describes the VQA task in natural language. According to the user input question, we filter the most similar image and question inputs from the training set, we calculate the cosine similarity of the fused features between the test input and each training input, and then select the top N nearest neighbors in the latent space as answer perception examples: ; Where M is the size of the local image and text dataset, N is the top N selected from M using argTOP, that is, the top N cosine similarity scores. Z refers to the image and question input by the user, z i refers to the image and question pairs in the local dataset.

[0059] The test input is the question entered by the user. Finally, the prompt is used to fill in the answer part of the prompt using the context learning ability of ChatGLM to obtain the predicted output. When a recommendation tendency is detected in the question and answer, the processed text and image information will be passed to the multimodal recommendation module. The speech recognition module is used to recognize the speech data input into the multimodal intelligent question-answering and recommendation system and process it into text form; specifically, a fully end-to-end speech recognition model structure based on Transformer is used; The speech recognition module specifically performs the following steps: Step B1: preprocessing the original audio data; Extract the audio feature FBank of the original audio. At the same time, preprocess the transcribed text corresponding to the original audio, convert the text content into discrete numbers, form a character index sequence, form the dictionary index corresponding to each character, and obtain the feature vector of the character index sequence through embedding; Step B2: Process the audio feature Fbank using a one-dimensional resnet structure, wherein the one-dimensional resnet structure includes 12 one-dimensional dilated causal convolutions with a convolution kernel size of 3 using residual links, to obtain audio features for speech recognition as input to the encoder; Step B3: Input the audio features for speech recognition into the encoder, further abstract the input features, and use 6 self-attention layers to learn context information to obtain intermediate features for decoding; Step B4: Use the transformer decoder structure and 6 cross-attention layers to train the intermediate features and encoder input. Map the output tensor to a vector of the same dimension as the character dictionary through a fully connected layer. Get the character in the dictionary corresponding to the index of the maximum value at each time step. In this process, in order to achieve parallel training, add the starting character before the input character index sequence. <sos>, and add the end character at the end of the character index sequence <eos>, and add a mask in the calculation process of cross attention to prohibit paying attention to the information after this time step, and use the structure of a fully end-to-end speech recognition model structure based on Transformer to implement speech recognition end-to-end.

[0060] The multimodal recommendation module is as follows Figure 2 As shown in the figure, it includes a graph convolution module, a data augmentation module and a cross-modal translation module; in visual question answering, all input data will first enter the multimodal question answering module for processing to obtain embedded representations of different modalities. If the content in the question answering is related to multimodal recommendation, then after processing, the obtained data will be sent to the multimodal recommendation module. In addition to the aforementioned data, the interaction matrix between users and items, as well as the number information of users and items, will also be passed to the multimodal recommendation module separately. The data that the multimodal recommendation module needs to input include the data of user-item interaction, the IDs of the users and items, and the multimodal embedded representation of the items.

[0061] The cross-modal translation module identifies whether the data passed into the multimodal recommendation module is missing a modality. If there is a modality missing, the cross-modal translation module completes the translation task; the modality missing is the lack of one or two of the three modalities of text, audio, and vision. The cross-modal translation module will learn the commonalities and characteristics of the same item in different modalities based on the representation of the passed-in modality by using the multi-layer perceptron model MLP, thereby using the existing modality to generate a representation of the missing modality, which is then passed into the graph convolution module. Assuming that the data passed into the multimodal recommendation module only includes the two modalities of text and audio, and the content of the visual modality is missing, the cross-modal translation module will use the two modalities of text and audio to generate a representation of the visual modality.

[0062] The graph convolution module learns the vector representation of users and items through the graph convolutional neural network according to the interaction matrix of users and items; because it is multimodal data, the same graph convolution operation is performed on different modalities, and finally the embedded representations of users and items in the three modalities of text, vision, and speech are obtained respectively; the embedded representations are fused through the multi-head attention algorithm to obtain the embedded representations of users and items after modal fusion and ; The data augmentation module performs reliability calculation on the original user-item interaction matrix to improve the diversity of recommendations. The reliability calculation is based on the user-item interaction matrix. G , add weights to the edges in the user-item interaction graph to obtain the calculated user-item interaction matrix G’ ; The user-item interaction matrix G In the matrix, if a user interacts with an item, the corresponding position in the matrix is ​​1, otherwise it is 0; the weight is calculated by calculating the similarity between items based on all items that each user has interacted with; according to the calculated user and item interaction matrix G’ At this time, in order to increase the diversity of recommendations, according to G’ To create a data augmentation graph ; The data augmentation graph is created Specifically, among users and items that have not interacted, a random edge is established for them, and the weight of this edge is the calculated reliability; the data augmentation graph is obtained , and then augment the graph for the data Perform graph convolution and multi-head attention operations to obtain new embedding representations of users and items, respectively recorded as and ; Since it is randomly generated, it adds diversified items to the original recommendation. At this time, by designing contrastive learning, the embedding representation of users and items obtained in the graph convolution module is and , learning from data augmented graphs Convolution and The above diverse information can improve the diversity of the final recommendation. Finally, the result of multimodal recommendation, that is, and The module after being sent is sent back to the multimodal question-answering module, which uses the recommended items to process it into a reply that the user can understand. If a voice reply is required later, it will be sent to the speech synthesis module for processing.

[0063] The multimodal recommendation module specifically performs the following steps: Step C1: Get the embedding representation of items of different modalities , randomly generated user embedding representation The interaction matrix between users and items is sent to the multimodal recommendation module. , representing the three modes of text, vision, and speech respectively; in the multimodal recommendation module, the data first enters the cross-modal translation module; the operations to be performed by the cross-modal translation module are as follows: Step C1.1: Modal feature decomposition; the cross-modal translation module embeds the input , decomposed into two parts: modal shared features and modality-specific features ; This division can ensure that subsequent translation operations can proceed smoothly.

[0064] The modalities share features Used to capture the commonalities between different modalities and provide cross-modal consistency; Modality specific features Used to express the unique features of a modality and preserve the uniqueness of the modality; Step C1.1.1: Modality shared feature learning: learn the modality shared features between different modalities through supervised contrastive learning loss function; The contrastive learning loss function is as follows: ;in, P ( i ) represents all items that belong to the same category or have similar characteristics as item i, A ( i ) represents all items in the current batch, a∈ A ( i ) represents the items in the current batch except item i itself; the representation of items is divided into modal shared features E c and modality-specific features E s Two parts, then use Will and After concatenation, the images are projected into an embedding space shared by multiple modalities. In this formula, z i and z p Although from different modalities, similarities in the shared space can be directly compared. i 、z a 、z p Both represent the use of a projection head to embed the specified object. and The content obtained by projecting into the shared embedding space, p represents the subscript of another item in the same category as i, and τ is a scalar temperature parameter used to control the degree of scaling of the dot product; Step C1.1.2: Modality-specific feature learning: To ensure that modality-specific features carry unique and complementary information, they are learned by maximizing the dissimilarity between modalities and modality-shared features using negative L2 distance as the L2 loss function. The L2 loss function is specifically: ;in, M is the number of different modes, and It means the i modality-specific and shared features of the modalities; Step C1.1.3: If adding the distance between modality-shared features and modality-specific features does not produce the best result, design a loss function to narrow the distance between the transformed embedding representation and the true embedding representation through a multi-layer perceptron to ensure the significance of the modality-specific features; the best result satisfies: the modality-specific features can accurately capture the unique information of each modality and are not confused with information shared by other modalities. Modality-shared features should extract common information between multiple modalities and have high discrimination capabilities. The shared features and specific features of the modalities should also be appropriately decoupled, but not completely independent, that is, a semantic connection can be established between the two through certain mechanisms (such as cross-modal translation modules); The loss function is specifically: i The following features are obtained through a multi-layer perceptron j The embedding representation under the modality; the loss function ensures that the transformed embedding representation is as consistent as possible with the real embedding representation. The details are as follows: ; Where: MLP is a multi-layer perceptron, which is used to translate known modalities into missing modalities. i and E j Representing embedded representations under different modalities; Step C1.2: Train the total loss function and divide the different modal embedding representations of the items into modality-shared features and modality-specific features by executing step C1.1; The total loss function is as follows: ; where α con , α dis , α trans is the learnable coefficient; After that, when a modality is missing, the trained model can be used to obtain the representation of the missing modality and send it to the subsequent module. When a modality is missing, it is sent to the cross-modal translation module to obtain the prediction and replacement of the missing modality and obtain the representation of the missing modality, as shown below: ; Step C2: Add the obtained embedding representations of all modalities of the item, i.e., modality-shared features and modality-specific features, to the user embedding representation , and are sent to the graph convolution module together; for different item modality embedding representations, the same graph convolution operation is performed on the original user-item interaction matrix G; the graph convolution operation uses the lightGCN algorithm, and the specific formula is as follows, recording the embedding representation obtained by each layer of convolution, and then averaging the k representations as the embedding representation of users and items at the current stage: ;in, Represents the embedding vector of node i in the kth layer of the user-item interaction matrix G. The embedding representation in the initial state is the embedding representation of items of different modalities obtained in the cross-modal translation module. For the embedding representation of items of different modalities, when performing graph convolution operations, the initial embedding representation of the user is randomly generated based on the user ID. represents the number of neighbors of node i, that is, the degree of the node on G; Step C3: Input the original user-item interaction matrix into the data augmentation module to perform reliability calculation; specifically, randomly sample the user-item pairs that are not in the original interaction matrix, add them to the original user-item interaction graph, and add reliability weights to generate a new interaction matrix based on the embedding representation of all modalities of the item plus the user embedding representation , in the data augmentation graph Perform graph convolution on the image to obtain a new embedding representation, and then use this new embedding representation to calculate a new loss function; The reliability calculation is specifically as follows: ; From the above formula, we can see that the reliability is calculated by calculating the cosine similarity.

[0065] The new loss function is as follows: ;in, and Respectively represented in the user-item interaction matrix G And data augmentation graph The user embedding representation obtained after convolution on . Note that since the convolution operation is performed on each modality, the loss function is actually calculated on each modality. U represents all users, u and v represent two different users in U, f() represents the cosine similarity function, and τ represents the temperature hyperparameter.

[0066] Step C4: The user and item embedding representations of different modalities obtained by the graph convolution module are sent to the multi-head attention model to fuse the multimodal information. The scores are calculated after multiplication based on the final fused user and item embedding representations. The item information with the highest score that has not yet interacted with the user is sent to the multimodal question-answering module.

[0067] Display the information of the items with the highest scores and which have not yet been interacted with by the user to the user. If text information and images are obtained, they are directly output. If they need to be translated into speech for output, the text data is sent to the speech synthesis module. The speech synthesis module includes an emotion vae sampling module, an emotion fusion module and an end-to-end decoder module; The emotion VAE sampling module includes an audio encoder, an emotion encoder and a decoder. The audio encoder is stacked by three layers of convolutional neural networks with residual links, which are used to obtain the mean and variance of the Mel-spectrogram. The Mel-spectrogram is obtained from an audio file. The audio is first sampled at a sampling rate of 22k, and then the short-time amplitude spectrum is obtained through pre-emphasis, framing, windowing and short-time Fourier transform. The short-time amplitude spectrum is passed through a Mel filter group to obtain a Mel-spectrogram. The emotion encoder is composed of a fully connected layer, which is used to obtain the mean and variance information in the emotion feature. The emotion feature is a vector corresponding to a discrete emotion label. The mean and variance of the audio spectrogram feature are calculated by KL divergence with the mean and variance of the emotion feature, and the mean and variance of the audio spectrogram feature are used to sample the emotion feature vector; wherein the KL divergence is used as part of the overall loss calculation of the speech synthesis module; the decoder is composed of two fully connected layers, which are used to linearly transform the sampled emotion features to increase the expressiveness and robustness of the model; The sentiment fusion module includes a diffusion model with a unet network as a Gaussian noise predictor and multiple kernel predictors. The diffusion model with a unet network as a Gaussian noise predictor uses a convolution-based sentiment embedding method in the downsampling process. This is a method for fusing sentiment features with text features through convolution operations. Similar to lvcnet, we do not split the sentiment features and the input, but directly perform convolution operations on the text features and the Gaussian noise required by the diffusion model. In the process of downsampling each layer of the unet network, a kernel predictor is designed. The kernel predictor first maps the sentiment feature dimension to a high-dimensional vector through a fully connected layer, and then transforms the high-dimensional vector into the shape of a pair of convolution kernels required for the parameters. A pair of convolution kernels perform convolution operations on the text features respectively, and then multiply the results to obtain the output of the fusion of the kernel predictor and the text features of this layer. In the process of three-layer downsampling in the unet network, one-dimensional convolutions with expansion coefficients of 1, 2, and 3 are used respectively. We believe that this method can effectively fuse sentiment features at different levels. This process is expressed as: { W f , W g }_ m =Kernel\Predictor(e), ; Where x represents the text feature, e represents the expanded sentiment feature, and Represents a pair of emotional convolution kernels generated simultaneously, z represents the feature after emotional fusion, and σ represents the relu activation function. Since emotional convolution can generate different kernels for different conditional sequences, it has a stronger ability to model long-term dependencies than traditional convolutional networks, which helps to integrate emotional information during model downsampling.

[0068] The end-to-end decoder module includes a generator module and a discriminator module; the generator module is used to convert the audio spectrogram into a time domain waveform, and includes four layers of convolutional neural networks with different expansion coefficients and residual links, and records the output results of each layer of the neural network; the discriminator module is used to perform sentiment classification and distinguish audio differences, and specifically has two training objectives, namely, the difference calculation between the synthetic audio and the real audio, and the difference between the sentiment classification label of the generated audio and the real label, wherein the difference calculation between the synthetic audio and the real audio is specifically: for the synthetic audio and the real audio, the generator module is used to calculate the output results of each layer of the neural network and the difference of the real audio after each layer of the neural network processing and the waveform difference with the real time domain audio, and the sentiment classification label of the generated audio is specifically: using the open source timnet model, the audio spectrogram is analyzed through a three-layer bidirectional convolutional neural network, and the difference in the results after the convolution operation of the real audio and the synthetic audio at different expansion coefficients is calculated; the adversarial training loss of the end-to-end decoder module is: , , , ; where x is the generated audio, y is the real audio, T represents the total number of discriminator layers, and D l Represents the feature map of the discriminator layer l, with N features l , k represents the dilation coefficient of the bidirectional causal convolution, is the expected formula after given x and y, is the expectation after given x, G() is the generator module, D() is the discriminator module, n is the number of convolutional layers with different dilation coefficients, is the sentiment classification loss in the discriminator in the decoder module, is the difference loss of each layer of the neural network in the generator module in the decoder module, is the adversarial loss of the generator and the discriminator in the decoder module.

[0069] The speech synthesis module is as follows Figure 3 As shown, perform the following steps: Step D1: Construct a single-person emotional speech dataset, extract the triples consisting of emotional speech, text, and emotional labels in the single-person emotional speech dataset, regularize the text, remove punctuation and meaningless characters, modify the numbers to the corresponding English expressions, and set the discrete emotional labels to five integers of 0, 1, 2, 3, and 4, representing anger, sadness, neutrality, happiness, and surprise, respectively; Step D2: Convert the text in the triple into a phoneme sequence using the g2p_en interface and obtain its corresponding phoneme feature vector; Step D3: convert the emotional speech in the triple into the corresponding mel spectrogram features; Step D4: Find the phoneme feature vector and obtain the phoneme feature through the text encoder. Since the length of the phoneme feature is much smaller than the length of the mel spectrogram, use the monotone alignment search method to find the path from the one-to-many phoneme feature to the mel spectrum length with the smallest difference. Then, expand the phoneme feature sequence according to the duration required for each phoneme feature to make it consistent with the mel spectrogram length. Step D5: Use VAE to implement low-dimensional continuous space sentiment modeling, find the sentiment vector corresponding to the sentiment label in the triple, and obtain the mean and variance of the low-dimensional feature distribution through the prior encoder, where the prior encoder consists of two layers of fully connected layers, uses the reparameterization technique to sample in the low-dimensional latent space, and generates the corresponding sentiment features through the decoder. The decoder is also implemented using two layers of fully connected layers to obtain the sentiment features to guide the generation model; Step D6: The phoneme feature sequence with the same length obtained in step D4 and the mel spectrogram after adding Gaussian noise are stacked, and then input into the diffusion probability model as a 2-channel plane tensor. The diffusion probability model uses a Gaussian noise predictor with a unet structure to reconstruct the input. In each downsampling layer of unet, it includes a layer of emotional convolution module, a layer of ordinary convolution module and a layer of downsampling layer, wherein the emotional convolution module includes a layer of emotional convolution and two layers of ordinary convolution, and the ordinary convolution module includes two layers of ordinary convolution. The convolution kernels used in the ordinary convolution operation are all of size 3×3, and the emotional convolution kernels are all of size 2×3. In the convolution, the emotional features are expanded according to the number of input channels × the number of convolution kernel parameters × the number of output channels, and then two layers of ordinary convolution operations are performed to improve their expressiveness and convert them into 3×3 emotional convolution kernels; the generated emotional convolution kernel is used to perform convolution operations on the 2-channel plane tensor to obtain the feature tensor to be downsampled; then the ordinary convolution operation is performed to obtain the downsampled output of this layer; since a three-layer unet network is used for training, in the process of downsampling each layer, the emotional convolution kernel uses an increasing expansion coefficient to increase the receptive field, so as to achieve modeling of factor feature sequences of different ranges and levels. The diffusion probability model is as follows: Figure 5 As shown in the figure, a phoneme feature sequence and a mel spectrogram with Gaussian noise added are used as input, and the mel spectrogram is used as output. At the same time, a decoder is trained, which takes the mel spectrogram obtained by processing real audio as input and the time domain audio waveform as output. The decoder uses a hifigan structure and adds an emotional speech classification module to the discriminator module. The emotional speech classification module uses a bidirectional convolution with three layers of different expansion coefficients to extract the emotional features of the synthesized audio and the real audio, and guides the generator to express the emotion in the speech by distinguishing the emotional difference between the generated audio and the real audio.

[0070] Step D7: Finally, the voice is output to the user, where the output includes not only the voice information but also the corresponding text information and pictures.

[0071] The above description is only a preferred embodiment of the present disclosure and an explanation of the technical principles used. Those skilled in the art should understand that the scope of the invention involved in the embodiments of the present disclosure is not limited to the technical solutions formed by a specific combination of the above technical features, but should also cover other technical solutions formed by any combination of the above technical features or their equivalent features without departing from the above inventive concept. For example, the above features are replaced with (but not limited to) technical features with similar functions disclosed in the embodiments of the present disclosure to form a technical solution.< / eos> < / sos> < / eos> < / sos>

Claims

1. A multimodal intelligent question-answering and recommendation system supporting emotional speech output, characterized in that: Specifically, it includes a multimodal question-answering module, a multimodal recommendation module, a speech recognition module, and a speech synthesis module; The multimodal question answering module includes: a question-oriented image description generation module, an external knowledge context expansion module and a question-aware visual question answering module; The question-oriented image description generation module is used to convert the image input by the user into a natural language form; the module uses the BLIP2 model to receive the image, the question and the instruction with the user input, locates the corresponding object from the image according to the input question, and outputs the natural language of the image description oriented to the user input; The external knowledge context extension module provides common sense knowledge, and uses the Comet knowledge base COMET to provide common sense knowledge; the Comet knowledge base COMET contains entities and relationships between entities, and focuses on social common sense centered on events, including the motivations and influences of event participants; the motivation refers to the internal driving force that determines the behavior of the participants; the influence refers to the changes that the behavior of the participants will cause to other events and entities; The question-aware visual question-answering module is used to complete the final question answering. The module uses the contextual learning ability of Zhipu Qingyan AI large model ChatGLM for visual question answering, and specifically includes two components: an answer reasoning generation component and a heuristic enhancement prompt component; the answer generation component generates two types of answer reasoning, namely, answer candidate sets and answer-aware examples; the heuristic enhancement prompt component integrates heuristic answers, questions, and question-oriented image descriptions into formatted prompts to instruct Zhipu Qingyan AI large model ChatGLM to predict answers; The speech recognition module is used to recognize the speech data input into the multimodal intelligent question-answering and recommendation system and process it into text form; specifically, a fully end-to-end speech recognition model structure based on Transformer is used; The multimodal recommendation module includes a graph convolution module, a data augmentation module and a cross-modal translation module; The cross-modal translation module identifies whether the data passed into the multimodal recommendation module is missing modality. If so, the cross-modal translation module completes the translation task. The modality missing refers to the lack of one or two of the three modalities of text, audio, and vision. The cross-modal translation module learns the commonalities and characteristics of the same item in different modalities based on the representation of the passed-in modality by using the multi-layer perceptron model MLP, thereby using the existing modality to generate the representation of the missing modality, which is then passed into the graph convolution module. The graph convolution module learns the vector representation of users and items through the graph convolutional neural network according to the interaction matrix of users and items; performs the same graph convolution operation on different modalities, and finally obtains the embedded representation of users and items in the three modalities of text, vision, and speech respectively; fuses the embedded representation through the multi-head attention algorithm to obtain the embedded representation of users and items after modal fusion E u and E i ; The data augmentation module performs reliability calculation on the original user-item interaction matrix; the so-called reliability calculation is based on the user-item interaction matrix G , add weights to the edges in the user-item interaction graph to obtain the calculated user-item interaction matrix G’ ; The user-item interaction matrix G In the matrix, if a user interacts with an item, the corresponding position in the matrix is ​​1, otherwise it is 0; the weight is calculated by calculating the similarity between items based on all items that each user has interacted with; according to the calculated user and item interaction matrix G’ , create a data augmentation graph ; The data augmentation graph is created Specifically, among users and items that have not interacted, a random edge is established for them, and the weight of this edge is the calculated reliability; the data augmentation graph is obtained , and then augment the graph for the data Perform graph convolution and multi-head attention operations to obtain new embedding representations of users and items, respectively recorded as and ; At this time, by designing contrastive learning, the embedding representation of users and items obtained in the graph convolution module is and , learning from data augmented graphs Convolution and The above diverse information finally presents the result of multimodal recommendation, namely and After being sent to the module, it is sent back to the multimodal question-answering module; The speech synthesis module includes an emotion vae sampling module, an emotion fusion module and an end-to-end decoder module; The emotion VAE sampling module includes an audio encoder, an emotion encoder and a decoder. The audio encoder is stacked by three layers of convolutional neural networks with residual links, which are used to obtain the mean and variance of the Mel-spectrogram. The Mel-spectrogram is obtained from an audio file. The audio is first sampled at a sampling rate of 22k, and then a short-time amplitude spectrum is obtained through pre-emphasis, framing, windowing and short-time Fourier transform. The short-time amplitude spectrum is passed through a Mel filter group to obtain a Mel-spectrogram. The emotion encoder is composed of a fully connected layer, which is used to obtain the mean and variance information in the emotion feature. The emotion feature is a vector corresponding to a discrete emotion label. The mean and variance of the audio spectrogram feature are calculated by KL divergence with the mean and variance of the emotion feature, and the mean and variance of the audio spectrogram feature are used to sample the emotion feature vector; wherein the KL divergence is used as part of the overall loss calculation of the speech synthesis module; the decoder is composed of two fully connected layers, which are used to linearly transform the sampled emotion features; The emotion fusion module includes a diffusion model with a unet network as a Gaussian noise predictor and multiple kernel predictors. The diffusion model with the unet network as a Gaussian noise predictor uses a convolution-based emotion embedding method in the downsampling process, and performs convolution operations on text features and Gaussian noise required by the diffusion model. A kernel predictor is designed in the process of downsampling each layer of the unet network. The kernel predictor first maps the emotion feature dimension into a high-dimensional vector through a fully connected layer, and then transforms the high-dimensional vector into the shape of a pair of convolution kernels required for parameters. A pair of convolution kernels perform convolution operations on text features respectively, and then multiply the results to obtain the output of the fusion of the kernel predictor and the text features of this layer. In the process of three-layer downsampling in the unet network, one-dimensional convolutions with expansion coefficients of 1, 2, and 3 are used respectively. This process is expressed as: , ; Where x represents the text feature, e represents the expanded sentiment feature, and represents a pair of emotion convolution kernels generated simultaneously, z represents the feature after emotion fusion, and σ represents the relu activation function; The end-to-end decoder module includes a generator module and a discriminator module; the generator module is used to convert the audio spectrogram into a time domain waveform, and includes four layers of convolutional neural networks with different expansion coefficients and residual links, and records the output results of each layer of the neural network; the discriminator module is used to perform sentiment classification and distinguish audio differences, and specifically has two training objectives, namely, the difference calculation between the synthetic audio and the real audio, and the difference between the sentiment classification label of the generated audio and the real label, wherein the difference calculation between the synthetic audio and the real audio is specifically: for the synthetic audio and the real audio, the generator module is used to calculate the output results of each layer of the neural network and the difference of the real audio after each layer of the neural network processing and the waveform difference with the real time domain audio, and the sentiment classification label of the generated audio is specifically: using the open source timnet model, the audio spectrogram is analyzed through a three-layer bidirectional convolutional neural network, and the difference in the results after the convolution operation of the real audio and the synthetic audio at different expansion coefficients is calculated; the adversarial training loss of the end-to-end decoder module is: , , , ; where x is the generated audio, y is the real audio, T represents the total number of discriminator layers, and D l Represents the feature map of the discriminator layer l, with N features l , k represents the dilation coefficient of the bidirectional causal convolution, is the expected formula after given x and y, is the expectation after given x, G() is the generator module, D() is the discriminator module, n is the number of convolutional layers with different dilation coefficients, is the sentiment classification loss in the discriminator in the decoder module, is the difference loss of each layer of the neural network in the generator module in the decoder module, is the adversarial loss of the generator and the discriminator in the decoder module.

2. According to claim 1, a multimodal intelligent question-answering and recommendation system supporting emotional speech output is characterized in that: The answer candidate set includes several groups of answers, and for each answer perception example, a group of answers is generated using the reasoning ability of ChatGLM; The answer perception example is: according to the image and question combination input by the user, the most similar image-question pair is calculated from the VQA dataset using cosine similarity as the answer perception example; Heuristic answers include answer-aware examples, answer candidate sets, and task instructions.

3. The multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 1, characterized in that: The multimodal question-answering module specifically performs the following steps: Step A1: The question-oriented image description generation module receives images and questions input by the user and outputs image descriptions, treating the existing VQA2.0 dataset as a natural source of tasks and task-related visual detail pairs; synthesizing question-aware captions by combining image captions and question-answer pairs using the contextual learning of the Zhipu Qingyan AI model ChatGLM; the prompts of the Zhipu Qingyan AI model ChatGLM contain task instructions, human-written examples, and VQA question-image pairs for synthesizing task-aware captions; the examples include questions, image descriptions, and answers; Step A1.1: Generate an image description training set using ChatGLM, where the image description dataset contains images and descriptions; each image is represented by concatenating 5 human-written COCO captions; Step A1.2: Extract 5 candidate captions for each question-answer pair from ChatGLM; design a pipeline to filter out the best candidate captions as training samples for the question-oriented image description generation module; For each candidate subtitle, the soft VQA accuracy is used to compare the answer corresponding to the candidate subtitle with the true answer to softly score the candidate subtitle; The soft VQA accuracy is obtained by the character error rate, and the candidate subtitles are sorted according to the soft score, and the candidate subtitles with the highest soft score are selected to generate training samples; The soft accuracy is obtained by the following formula: ; where g i refers to groundtruth, which is the standard answer, a refers to answer, which is the answer given by the system, n is the number of standard answer sets for each question, x, y, z refer to the results calculated based on CER, and the current result is the data in the answer set. The lowest three are taken and identified as x, y, z, and CER is the character error rate; Step A1.3: For the question-oriented image description generation module, the training samples obtained in step A1.2 are reassembled into instruction prompts in the form of question-title pairs using the BLIP2 model; Step A2: Introduce external common sense knowledge for the visual question answering task through the external knowledge context extension module, use the knowledge base generation model COMET to generate contextualized common sense knowledge related to the question and image content, and filter and extract the required reasoning; Step A2.1: Generate common sense reasoning using the COMET model. First, the COMET model performs language processing on the question and converts it into a declarative sentence. Then, the object detection model YOLO is used to extract object labels from the image and combine the object labels with the question to form a question object phrase. Step A2.1.1: Use the COMET model to generate common sense reasoning based on the input phrase; Step A2.1.2: Clean up repeated or redundant inferences by detecting vocabulary overlap and removing sentences with similarity greater than a set threshold; Step A2.2: After generating the inferences, select the most useful information from them. Specifically, use SBERT to encode them in the same vector space, sort them by cosine similarity, and select only the top 5 most relevant inferences by calculating the problem and each inference; Step A3: The question-aware visual question answering module uses a portion of the VQA examples as task examples for ChatGLM and concatenates the contextual learning examples together to form a prompt; each contextual learning example contains a question, a generated image description, and an answer; the user-input question is appended to the contextual learning example and input to ChatGLM; ChatGLM generates a predicted answer based on an open text generation method.

4. The multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 1, characterized in that: The speech recognition module specifically performs the following steps: Step B1: preprocessing the original audio data; Extract the audio feature FBank of the original audio. At the same time, preprocess the transcribed text corresponding to the original audio, convert the text content into discrete numbers, form a character index sequence, form the dictionary index corresponding to each character, and obtain the feature vector of the character index sequence through embedding; Step B2: Process the audio feature Fbank using a one-dimensional resnet structure, wherein the one-dimensional resnet structure includes 12 one-dimensional dilated causal convolutions with a convolution kernel size of 3 using residual links, to obtain audio features for speech recognition as input to the encoder; Step B3: Input the audio features for speech recognition into the encoder, further abstract the input features, and use 6 self-attention layers to learn context information to obtain intermediate features for decoding; Step B4: Use the transformer decoder structure and 6 cross-attention layers to train the intermediate features and encoder input. Map the output tensor to a vector of the same dimension as the character dictionary through a fully connected layer. Get the character in the dictionary corresponding to the index of the maximum value at each time step. Specifically, add the starting character before the input character index sequence. <sos>, and add the end character at the end of the character index sequence <eos> , and add masks in the calculation process of cross attention, using the structure of a fully end-to-end speech recognition model based on Transformer to achieve speech recognition end-to-end.< / eos> < / sos> 5. The multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 1, characterized in that: The multimodal recommendation module specifically performs the following steps: Step C1: Get the embedding representation of items of different modalities , randomly generated user embedding representation The interaction matrix between users and items is sent to the multimodal recommendation module. , representing the three modalities of text, vision, and speech respectively; in the multimodal recommendation module, the data first enters the cross-modal translation module; Step C2: Add the obtained embedding representations of all modalities of the item, i.e., modality-shared features and modality-specific features, to the user embedding representation , and are sent to the graph convolution module together; for different item modality embedding representations, the same graph convolution operation is performed on the original user-item interaction matrix G; the graph convolution operation uses the lightGCN algorithm, and the specific formula is as follows, recording the embedding representation obtained by each layer of convolution, and then averaging the k representations as the embedding representation of users and items at the current stage: ;in, Represents the embedding vector of node i in the kth layer of the user-item interaction matrix G. The embedding representation in the initial state is the embedding representation of items of different modalities obtained in the cross-modal translation module. For the embedding representation of items of different modalities, when performing graph convolution operations, the initial embedding representation of the user is randomly generated based on the user ID. The table represents the number of neighbors of node i, that is, the degree of the node on G; Step C3: Input the original user-item interaction matrix into the data augmentation module to perform reliability calculation; specifically, randomly sample the user-item pairs that are not in the original interaction matrix, add them to the original user-item interaction graph, and add reliability weights to generate a new interaction matrix based on the embedding representation of all modalities of the item plus the user embedding representation , in the data augmentation graph Perform graph convolution on the image to obtain a new embedding representation, and then use this new embedding representation to calculate a new loss function; Step C4: The user and item embedding representations of different modalities obtained by the graph convolution module are sent to the multi-head attention model to fuse the multimodal information. The scores are calculated after multiplying the embedded representations of the fused user and item, and the item information with the highest score that has not yet interacted with the user is sent to the multimodal question-answering module. The information of items with the highest scores and that have not yet been interacted with by the user is displayed to the user. If text information and images are obtained, they are directly output. If they need to be translated into speech for output, the text data is sent to the speech synthesis module.

6. A multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 5, characterized in that: The step C1 specifically comprises the following steps: Step C1.1: Modal feature decomposition; the cross-modal translation module embeds the input , decomposed into two parts: modal shared features and modality-specific features ; The modalities share features Used to capture commonalities between different modalities; Modality specific features Used to express the unique characteristics of a modality; Step C1.1.1: Modality shared feature learning: learn the modality shared features between different modalities through supervised contrastive learning loss function; The contrastive learning loss function is as follows: ;in, P ( i ) represents all items that belong to the same category or have similar characteristics as item i, A ( i ) represents all items in the current batch, a∈ A ( i ) represents the items in the current batch except item i itself; the representation of items is divided into modal shared features and modality-specific features Two parts, then use Will and After concatenation, it is projected into the embedding space shared by multiple modalities, z i 、z p 、z p Both represent the use of a projection head to embed the specified object. and The content obtained by projecting into the shared embedding space, p represents the subscript of another item in the same category as i, and τ is the scalar temperature parameter; Step C1.1.2: Modality-specific feature learning: learning is performed by maximizing the dissimilarity between modalities and modality-shared features, using negative L2 distance as the L2 loss function; The L2 loss function is specifically: ;in, M is the number of different modes, and It means the i modality-specific and shared features of the modalities; Step C1.1.3: If adding the distance between the modality-shared features and the modality-specific features does not produce the best results, design a loss function to narrow the distance between the transformed embedding representation and the true embedding representation through a multi-layer perceptron; The loss function is specifically: i The following features are obtained through a multi-layer perceptron j The embedding representation under the modality is as follows: ; Where: MLP is a multi-layer perceptron, E i and E j Representing embedded representations under different modalities; Step C1.2: Train the total loss function and divide the different modal embedding representations of the items into modality-shared features and modality-specific features by executing step C1.1; The total loss function is as follows: ; where α con , α dis , α trans is the learnable coefficient; When a modality is missing, it is sent to the cross-modal translation module to obtain the prediction and replacement of the missing modality, and the representation of the missing modality is obtained as shown below: .

7. The multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 5, characterized in that: The reliability calculation in step C3 is specifically as follows: ; The new loss function is as follows: ;in, and Respectively represented in the user-item interaction matrix G And data augmentation graph The user embedding representation obtained after convolution on , U represents all users, u and v represent two different users in U, f() represents the cosine similarity function, and τ represents the temperature hyperparameter.

8. The multimodal intelligent question-answering and recommendation system supporting emotional speech output according to claim 1, characterized in that: The speech synthesis module specifically performs the following steps: Step D1: Construct a single-person emotional speech dataset, extract the triples consisting of emotional speech, text, and emotional labels in the single-person emotional speech dataset, regularize the text, remove punctuation and meaningless characters, modify the numbers to the corresponding English expressions, and set the discrete emotional labels to five integers of 0, 1, 2, 3, and 4, representing anger, sadness, neutrality, happiness, and surprise, respectively; Step D2: Convert the text in the triple into a phoneme sequence using the g2p_en interface and obtain its corresponding phoneme feature vector; Step D3: convert the emotional speech in the triple into the corresponding mel spectrogram features; Step D4: Find the phoneme feature vector, obtain the phoneme feature through the text encoder, use the monotone alignment search method to find the path from the one-to-many phoneme feature to the mel spectrum length with the smallest difference, and then expand the phoneme feature sequence according to the duration required for each phoneme feature obtained to make it consistent with the mel spectrogram length; Step D5: Use VAE to implement low-dimensional continuous space sentiment modeling, find the sentiment vector corresponding to the sentiment label in the triple, and obtain the mean and variance of the low-dimensional feature distribution through the prior encoder, where the prior encoder consists of two layers of fully connected layers, uses the reparameterization technique to sample in the low-dimensional latent space, and generates the corresponding sentiment features through the decoder. The decoder is also implemented using two layers of fully connected layers to obtain the sentiment features to guide the generation model; Step D6: stack the phoneme feature sequences with the same length obtained in step D4 and the mel spectrogram after adding Gaussian noise, and then input them into the diffusion probability model as a 2-channel plane tensor. The diffusion probability model uses a Gaussian noise predictor with a unet structure to reconstruct the input. In each downsampling layer of unet, it includes a layer of emotional convolution module, a layer of ordinary convolution module and a layer of downsampling layer, wherein the emotional convolution module includes a layer of emotional convolution and two layers of ordinary convolution, and the ordinary convolution module includes two layers of ordinary convolution. The convolution kernels used in the ordinary convolution operation are all of a size of 3×3. In the emotional convolution, the emotional features are expanded according to the number of input channels × the number of convolution kernel parameters × the number of output channels, and then converted into a 3×3 emotional convolution kernel after two layers of ordinary convolution; The generated emotional convolution kernel is used to perform a convolution operation on the 2-channel plane tensor to obtain the feature tensor to be downsampled; then, the ordinary convolution operation is performed to obtain the downsampled output of this layer; the diffusion probability model takes the phoneme feature sequence and the mel spectrogram with Gaussian noise added as input, and takes the mel spectrogram as output. At the same time, a decoder is trained, which takes the mel spectrogram obtained by real audio processing as input and the time domain audio waveform as output. The decoder uses a hifigan structure, and an emotional speech classification module is added to the discriminator module. The emotional speech classification module uses a bidirectional convolution containing three layers of different expansion coefficients to extract the emotional features of the synthesized audio and the real audio, and guides the generator to express the emotion in the speech by distinguishing the emotional difference between the generated audio and the real audio; Step D7: Finally, the voice is output to the user, where the output includes not only the voice information but also the corresponding text information and pictures.

Images