Multi-modal natural language understanding and generating system and method

Abstract

The invention discloses a multi-modal natural language understanding and generating system and method. The method comprises the following steps: constructing a cross-modal pre-training module, training a multi-modal encoder, and establishing a cross-modal association mapping space; mixing prompt fine tuning is carried out, and a complete blank filling template is constructed; according to the intention reasoning network, extracting multi-round dialogue intention representation of the user, and retrieving an external knowledge base for fine-grained reasoning; constructing a unified semantic representation framework, embedding the text, the image and the voice into a unified space, and generating a query vector of multi-modal intention perception; and the knowledge query module based on key value memory generates entity-level multi-modal replies and optimizes the semantic comprehension and generation capability of the dialogue model. According to the method, the multi-modal information understanding and generating capacity is improved, deep association and understanding of image and text information are achieved, downstream task adaptability is enhanced, task completion accuracy and efficiency are improved, unified semantic representation of the multi-modal information is achieved, and support is provided for information retrieval and utilization.

Description

Technical Field

[0001] The present invention relates to the field of natural language processing, and in particular to a multimodal natural language understanding and generation system and method. Background Art

[0002] In the field of natural language processing, with the continuous advancement of artificial intelligence technology, multimodal natural language understanding and generation technologies have gradually become a research frontier. Traditional natural language processing methods mainly focus on a single text modality, such as processing only written or spoken text data. Although this approach has been successful in certain specific tasks, it often ignores information from images, speech, and other non-text modalities, which are crucial for understanding human communication in the real world. For example, on social media, users may express their emotions and opinions through a variety of forms such as text and images. Therefore, single-modality processing methods have limitations in capturing and understanding this complex information.

[0003] As technology develops, researchers are exploring how to integrate information from different modalities to more fully understand language and context. Multimodal NLP technology attempts to combine multiple data sources, such as text, images, and sound, to provide richer and more accurate information processing capabilities. For example, in a given scene, a picture with a text description can provide more information than text or images alone. Multimodal systems can analyze this information simultaneously to better understand user intent and emotions, providing a more humane interactive experience. This technology has broad application prospects in areas such as intelligent assistants, self-driving cars, medical diagnosis, and education, and can greatly enhance the understanding and generation capabilities of artificial intelligence systems.

[0004] The patent name is a solution of a multimodal RAG knowledge question-answering method and device applied to vertical fields, which supports multiple modal text inputs to be answered, covering text, pictures, and audio, so as to make full use of the rich semantic information of multimodal data. By using multiple preset retrieval models, the method can retrieve knowledge fragments related to the text to be answered and obtain diversified retrieval results. Next, the RRF algorithm is used to fuse these retrieval results, and the scores are calculated through the target reranking model to achieve the re-ranking of the results. On this basis, the sorting results are further fine-tuned in combination with the search content and the preset knowledge frequency information to ensure that they are more accurate and meet the expectations of users. The natural language understanding ability of the large language model, the target reward model, and the relevant information of RAG recall are combined to generate the optimal answer data. At the same time, the target scoring model and the knowledge of domain experts are used to provide more professional and high-quality answer data that meets actual needs.

[0005] As can be seen, with the continuous development of artificial intelligence technology, multimodal natural language understanding and generation technology has gradually become a research hotspot. In recent years, researchers have begun to explore multimodal natural language understanding and generation technology. This technology aims to improve the accuracy and efficiency of natural language processing by integrating information from different modalities. However, existing multimodal natural language understanding and generation methods still face many challenges. For example, how to establish an effective associative mapping space between different modalities to achieve cross-modal information fusion; how to effectively incorporate human prior knowledge into downstream tasks to alleviate language bias. Summary of the Invention

[0006] The purpose of the present invention is to overcome the shortcomings of the existing technology and provide a multimodal natural language understanding and generation system and method, establish an effective associative mapping space between different modalities to achieve cross-modal information fusion; effectively integrate human prior knowledge into the task to alleviate the problem of language bias.

[0007] The object of the present invention is achieved through the following technical solutions:

[0008] A method based on multimodal natural language understanding and generation, including the following contents:

[0009] S1: Construct a cross-modal pre-training module to train a multimodal encoder on an image-text dataset through self-supervised learning and establish a cross-modal associative mapping space;

[0010] S2: Perform hybrid prompt fine-tuning to build cloze templates for downstream tasks, incorporating trainable prompt slots and human prior knowledge, and mitigate language bias through adaptive topic classification and comparative example filling;

[0011] S3: Based on the intent reasoning network, combined with a memory mechanism and a hierarchical reply strategy, it extracts intent representations from multiple rounds of user conversations, retrieves external knowledge bases, and performs fine-grained reasoning;

[0012] S4: Build a unified semantic representation framework to embed text, images, and speech into a unified space. Fine-tune through image-text matching and word-region alignment tasks to generate multimodal intent-aware query vectors.

[0013] S5: A knowledge query module based on key-value memory generates entity-level multimodal responses and optimizes the semantic understanding and generation capabilities of the dialogue model through combinatorial fine-tuning.

[0014] As a preferred embodiment, the cross-modal pre-training module in S1 specifically includes:

[0015] S11: Extract common entities of image-text pairs through anchor detection and construct anchor set;

[0016] S12: A cross-modal associative learning strategy is adopted to model fine-grained semantic associations between modalities in the implicit mapping space through mask region reconstruction, cross-modal contrast loss, and associative map classification tasks.

[0017] S13: Use the Transformer backbone network to perform multimodal feature interaction and output a joint representation that integrates vision and text.

[0018] As a preferred embodiment, the mixed prompt fine-tuning in S2 includes:

[0019] S21: Convert the question-answering task into a cloze template, rewrite the question into a declarative sentence using language reorganization rules, and insert trainable slots [ANS];

[0020] S22: Introducing a topic classification module to dynamically fill the [TOP] slot based on the user's historical conversations, narrowing the answer search scope;

[0021] S23: Using contrastive learning strategy, retrieve positive and negative examples from the knowledge base to fill the [S1]-[S3] slots and generate candidate answer neighborhoods.

[0022] As a preferred embodiment, the intention inference network in S3 includes:

[0023] S31: Extracting coarse-grained candidate knowledge from external knowledge base through memory network;

[0024] S32: Use a multi-hop attention mechanism to analyze user intent and generate an intent vector as a query condition;

[0025] S33: Combined with knowledge graph embedding, logical reasoning is performed on candidate knowledge and fine-grained reasoning paths are output.

[0026] As a preferred embodiment, the unified semantic representation framework described in S4 is implemented as follows:

[0027] S41: Use a multimodal embedder to encode text, images, and speech into vectors of the same dimension;

[0028] S42: Optimizing global semantic alignment via image-text matching tasks;

[0029] S43: Enhancing local entity-level associations through a word-region alignment task to generate user intent-aware query vectors.

[0030] As a preferred embodiment, the combined fine-tuning strategy in S5 includes:

[0031] S51: Jointly optimize semantic understanding loss, knowledge reasoning loss, and generation loss;

[0032] S52: A curriculum learning mechanism is used for three-stage training: first fine-tuning the modality alignment module, then optimizing the intent inference network, and finally jointly training the generation module.

[0033] S53: Constraining generation results through adversarial discriminant networks to improve response diversity and knowledge consistency.

[0034] As a preferred method, the multimodal data enhancement step is also included:

[0035] S61: Add noise perturbation to the speech data and convert it into text;

[0036] S62: Perform random cropping and color transformation on the image data to generate enhanced multimodal training samples.

[0037] Multimodal natural language understanding and generation system, including the following modules:

[0038] A pre-training module for training a multimodal encoder on image-text data through cross-modal associative learning to establish fine-grained modality alignment;

[0039] The fine-tuning module, which includes a hybrid prompt engine and a topic classifier, adapts to downstream tasks through trainable slots and injects human prior knowledge;

[0040] The intent reasoning module integrates a memory network and a multi-hop attention mechanism to achieve fine-grained retrieval and reasoning from user conversations to external knowledge;

[0041] The semantic fusion module uses a unified embedder to map text, images, and speech into the same semantic space and generate intent-aware query vectors;

[0042] The generation module outputs multimodal responses that match user intent based on a key-value memory library and adversarial training strategy.

[0043] As a preferred embodiment, the pre-training module further includes:

[0044] Anchor detection unit, which extracts common entities from image-text pairs as alignment anchors;

[0045] Associative Mapping Unit, which optimizes modality alignment via mask reconstruction, contrastive loss, and a cross-modal associative classification (AMC) task;

[0046] The feature interaction unit uses a multi-layer Transformer to achieve cross-modal attention interaction.

[0047] As a preferred embodiment, the fine-tuning module includes:

[0048] Template generator, which converts questions into cloze templates with [ANS] slots;

[0049] The sample filler retrieves positive and negative samples from the knowledge base to fill the [S1]-[S3] slots;

[0050] The bias correction unit constrains the model to rely on visual features to generate answers through contrastive loss.

[0051] The present invention has at least the following beneficial effects: the present invention improves the ability to understand and generate multimodal information. By constructing a cross-modal pre-training module, it is possible to effectively train a multimodal encoder on an image-text dataset and establish a cross-modal associative mapping space, thereby achieving deep association and understanding of image and text information. In addition, we have also enhanced the adaptability of downstream tasks. Through hybrid prompt fine-tuning technology, it is possible to construct a cloze template for specific downstream tasks, and incorporate trainable prompt slots and human prior knowledge, effectively alleviating the language bias problem and improving the accuracy and efficiency of task completion. A unified semantic representation of multimodal information is achieved. By embedding text, images, speech, etc. into a unified space, and performing fine-tuning of image-text matching and word-region alignment tasks, this method can generate multimodal intent-aware query vectors, providing strong support for information retrieval and utilization. BRIEF DESCRIPTION OF THE DRAWINGS

[0052] To reveal the technical details of the embodiments of the present invention, the following is a brief introduction to the drawings involved in the embodiments. It should be emphasized that these drawings only illustrate several embodiments of the present invention and should not be considered as defining the scope of the invention. Those skilled in the art can deduce other relevant drawings based on these drawings without engaging in creative work.

[0053] Figure 1 Schematic diagram of a method flow in an embodiment of the present invention;

[0054] Figure 2 To achieve an intuitive diagram of the interaction between cross-modal feature cues and contextual attention;

[0055] Figure 3 Schematic diagram of the fine-tuning process for hybrid cues. DETAILED DESCRIPTION

[0056] The technical solution of the present invention will be further described in detail below with reference to the accompanying drawings, but the protection scope of the present invention is not limited to the following.

[0057] In the following, embodiments of the present disclosure are described in detail with the aid of accompanying drawings. However, please be aware that the present disclosure is not limited to the specific forms shown herein. Rather, it should be understood to encompass various variations, equivalents, and / or alternatives to the embodiments of the present disclosure. In describing the drawings, the same reference numerals will be used to indicate similar components.

[0058] It should be understood that while the following description provides extensive specific details intended to facilitate a comprehensive understanding of the example embodiments, those skilled in the art will appreciate that the example embodiments can be implemented without these specific details. For example, systems may be presented in block diagram form to avoid excessive detail that would obscure the clarity of the examples. In other cases, unnecessary details regarding well-known processes, structures, and techniques may be omitted to maintain clarity of the examples.

[0059] like Figure 1 As shown, a method based on multimodal natural language understanding and generation includes the following: S1: constructing a cross-modal pre-training module, training a multimodal encoder on an image-text dataset through self-supervised learning, establishing a cross-modal associative mapping space, and utilizing cross-modal feature cues and contextual attention interactions to achieve fine-grained modality alignment;

[0060] Specific methods for implementing cross-modal feature hints and contextual attention interactions (see Figure 2 ):

[0061] A111: Cross-modal feature hinting mechanism, including anchor detection and feature exchange: Key regions in the image (e.g., "goggles" and "swim ring") are extracted using an object detector (e.g., Faster R-CNN) and matched with entity words in the text (e.g., "goggles" and "swim ring") to construct a set of anchors. During training, anchors are randomly selected, and the image region features are exchanged with the corresponding text word vectors (e.g., the "goggles" region feature in the image is replaced with the "goggles" word vector in the text). The model infers the exchanged features from the context, forcing learning of fine-grained cross-modal alignment.

[0062] A112: Contextual attention interactions include: ① Mixed-modal input sequence: Image region features and text word vectors are concatenated into a unified sequence, such as [image region 1, image region 2, ..., text word 1, text word 2, ...]. ② Multi-head cross-modal attention: In the Transformer layer, the self-attention mechanism is used to capture intra-modal associations (such as the spatial relationship between image regions). The cross-attention mechanism dynamically calculates the interaction weights between the image and text (for example, the "goggles" region has a higher attention score for the word "goggles"). When processing the text word "color", the model automatically focuses on relevant areas in the image (such as color blocks of clothing) and achieves local alignment through attention weights.

[0063] A113: Fine-grained alignment supervision signals include mask reconstruction and contrastive learning optimization. Mask reconstruction randomly masks entity words (e.g., "goggles") in text or corresponding regions in images, requiring the model to predict the masked content using cross-modal context. Contrastive learning optimization constructs positive and negative sample pairs (e.g., positive sample: image "goggles" + text "goggles"; negative sample: image "goggles" + text "hat"), using contrastive loss to narrow the representational distance between positive samples and push negative samples further away.

[0064] A114: Dynamic Theme Guide

[0065] Trainable prompt slots: Insert [ANS] (answer slot) and [TOP] (topic slot) into the input, such as rewriting the question "What is the girl wearing?" into the statement: "The girl is wearing [ANS]."

[0066] Adaptive filling: Dynamically fill the [TOP] slot based on conversation history or image content (e.g., the topic is "swimming equipment"), narrowing the answer search scope to relevant entities (e.g., "goggles," "swimsuits").

[0067] The following example illustrates

[0068] Input: Image (girl wearing swimming goggles) + question "What is on her eyes?"

[0069] Feature hint: The image anchor “goggle area” is detected to be associated with the text word “eyes”, and the features are exchanged to generate a mixed sequence.

[0070] Attention interaction: The model focuses the text “eyes” on the goggles area in the image through cross attention.

[0071] Alignment supervision: The masked text “goggles” is used, and the model predicts the word based on the image region and context, completing fine-grained alignment.

[0072] Output: The answer “goggles” is generated by the [ANS] slot and is aligned with the image region with high confidence.

[0073] During implementation, feature swapping forces the model to understand cross-modal semantic equivalence, rather than relying on unimodal statistical regularities. Attention weight visualization verifies that the model is focusing on the correct local regions (e.g., goggles rather than the background). Dynamic themes introduce human priors to prevent the model from fitting only to text labels (e.g., incorrectly associating "eyes" with "glasses"). This approach significantly improves the accuracy of image-text association in scenarios such as e-commerce recommendations and intelligent customer service.

[0074] S2: Perform hybrid prompt fine-tuning to build cloze templates for downstream tasks, incorporating trainable prompt slots and human prior knowledge, and mitigate language bias through adaptive topic classification and comparative example filling;

[0075] S3: Based on the intent reasoning network, combined with a memory mechanism and a hierarchical reply strategy, it extracts intent representations from multiple rounds of user conversations, retrieves external knowledge bases, and performs fine-grained reasoning;

[0076] S4: Build a unified semantic representation framework to embed text, images, and speech into a unified space. Fine-tune through image-text matching and word-region alignment tasks to generate multimodal intent-aware query vectors.

[0077] S5: A knowledge query module based on key-value memory generates entity-level multimodal responses and optimizes the semantic understanding and generation capabilities of the dialogue model through combinatorial fine-tuning.

[0078] This embodiment combines the memory mechanism and the hierarchical reply strategy, which can extract intent representations from multiple rounds of user conversations and retrieve external knowledge bases for fine-grained reasoning, so as to accurately understand user intent and generate responses that meet expectations. A unified semantic representation of multimodal information is achieved. By embedding text, images, voice, etc. into a unified space and fine-tuning the image-text matching and word-region alignment tasks, a multimodal intent-aware query vector can be generated, providing strong support for the retrieval and utilization of information. Supports the generation of entity-level multimodal replies. The knowledge query module based on key-value memory can generate entity-level multimodal replies containing rich information according to user needs, further improving user experience and satisfaction.

[0079] This embodiment achieves intelligent interaction in multimodal dialogue through multi-stage collaborative work. The present invention first establishes a foundation for cross-modal understanding and then gradually refines task capabilities. First, through pre-training on massive image and text data, the model learns to accurately associate image content with corresponding text (such as identifying "swimming goggles" in an image and associating it with a text description); then, task templates are used to guide the model to adapt to specific scenarios, incorporating human experience rules into question-answering tasks, narrowing the scope of answers through topic classification, and improving accuracy by comparing similar cases; then, an intent analysis network is constructed, combining conversation history to understand the user's deep needs, and filtering effective information from the knowledge base through multi-step reasoning; then, text, images, and voice are uniformly encoded, making different modal information comparable in the same semantic space, capable of both globally matching overall semantics and finely associating local details (such as binding the text "red skirt" to the corresponding area in the image); finally, the knowledge base is searched based on user intent, and a text-and-image combination answer is dynamically generated. The accuracy and diversity of the response are ensured through multi-task joint optimization. The entire process realizes the cognitive chain of observation, association, reasoning, and response, gradually solving key problems such as visual understanding bias, intention ambiguity, and complex knowledge association in multimodal dialogues.

[0080] In a preferred embodiment, the cross-modal pre-training module in S1 specifically includes:

[0081] S11: Extract common entities of image-text pairs through anchor detection and construct anchor set;

[0082] S12: A cross-modal associative learning strategy is adopted to model fine-grained semantic associations between modalities in the implicit mapping space through mask region reconstruction, cross-modal contrastive loss, and associative map classification (AMC) tasks.

[0083] S13: Use the Transformer backbone network to perform multimodal feature interaction and output a joint representation that integrates vision and text.

[0084] In a preferred embodiment, Figure 3 As shown, the hybrid prompt fine-tuning described in S2 includes:

[0085] S21: Convert the question-answering task into a cloze template, rewrite the question into a declarative sentence using language reorganization rules, and insert trainable slots [ANS];

[0086] S22: Introducing a topic classification module to dynamically fill the [TOP] slot based on the user's historical conversations, narrowing the answer search scope;

[0087] S23: Using contrastive learning strategy, retrieve positive and negative examples from the knowledge base to fill the [S1]-[S3] slots and generate candidate answer neighborhoods.

[0088] In a preferred embodiment, the intention inference network in S3 includes:

[0089] S31: Extracting coarse-grained candidate knowledge from external knowledge base through memory network;

[0090] S32: Use a multi-hop attention mechanism to analyze user intent and generate an intent vector as a query condition;

[0091] S33: Combined with knowledge graph embedding, logical reasoning is performed on candidate knowledge and fine-grained reasoning paths are output.

[0092] In a preferred embodiment, the unified semantic representation framework in S4 is implemented as follows:

[0093] S41: Use a multimodal embedder to encode text, images, and speech into vectors of the same dimension;

[0094] S42: Optimizing global semantic alignment via image-text matching tasks;

[0095] S43: Enhance local entity-level associations through the word-region alignment (WRA) task to generate user intent-aware query vectors.

[0096] In a preferred embodiment, the combined fine-tuning strategy in S5 includes:

[0097] S51: Jointly optimize semantic understanding loss, knowledge reasoning loss, and generation loss;

[0098] S52: A curriculum learning mechanism is used for three-stage training: first fine-tuning the modality alignment module, then optimizing the intent inference network, and finally jointly training the generation module.

[0099] S53: Constraining generation results through adversarial discriminant networks to improve response diversity and knowledge consistency.

[0100] The implementation method of knowledge query and combinatorial fine-tuning based on key-value memory includes: knowledge storage and query mechanism, multimodal response generation strategy and joint optimization training method. In terms of knowledge storage, the system encodes information such as product attributes and user historical conversations in the external knowledge base into a structured key-value pair format. Among them, the key (Key) is composed of the semantic vector of the entity or intent, such as the text embedding of "red dress"; the value (Value) is associated with multimodal data, including product image features, price text and user preference tags. For example, when the key is "red dress", the value may contain the image URL, price information (such as ￥299) and applicable scenarios (such as summer wear). The knowledge query process is divided into three steps: first, the intent inference network is used to analyze multiple rounds of user conversations to generate an intent vector; second, the cosine similarity between the vector and all keys in the knowledge base is calculated to retrieve the top-K related entities (in the knowledge query module, all candidate entities are sorted according to the similarity (such as cosine similarity) between the user intent vector and the entity key in the knowledge base, and the most relevant K entities are returned); finally, the candidate entities are logically inferred based on the knowledge graph embedding to generate a fine-grained answer list.

[0101] The multimodal response generation strategy dynamically selects the output format based on user intent. When a user requests an outfit recommendation, the system retrieves matching entities from the key-value memory and generates a mixed text and image response (e.g., displaying an image of a red chiffon dress with the caption "breathable, suitable for beach photos"). Simultaneously, speech synthesis technology provides a voice explanation of the sun protection factor. This generation process is implemented using a multimodal decoder: the text generation module extracts entity attributes from the knowledge base using a Transformer decoder; the image retrieval module selects the most relevant images through cross-modal matching; and the speech synthesis module converts the text response into a speech stream.

[0102] In terms of optimization strategy, the model uses a curriculum learning mechanism for phased fine-tuning: in the first phase, only the modality alignment module is optimized, enhancing global semantic alignment through image-text matching tasks; in the second phase, an intent inference network is introduced to improve knowledge retrieval capabilities; in the third phase, the generation module is jointly trained and an adversarial loss is added to improve response diversity. The joint loss function includes a semantic understanding loss (to optimize intent classification), a knowledge inference loss (to compare retrieval results with the ground truth), and a generation loss (to ensure text fluency and image-text consistency).

[0103] In order to optimize the semantic understanding, knowledge reasoning and generation capabilities of the multimodal dialogue model, a joint loss function is provided, which consists of three parts: semantic understanding loss (L understand )), knowledge reasoning loss (L reason ) and generation loss (L generate ). These three parts form the total loss through weighted summation, as follows:

[0104] Semantic understanding loss (L understand ), used to optimize the accuracy of user intent classification. Given a multimodal conversation history X (including text, images, etc.) input by the user, the model needs to predict its intent category distribution (C is the number of intent categories) and is aligned with the true label y through cross entropy loss:

[0105]

[0106] Among them, softmax(·) normalizes the prediction score to a probability distribution, y i is the true label of the i-th category (0 or 1, indicating whether it belongs to the category), The model's prediction score for the i-th category.

[0107] Knowledge Reasoning Loss (L reason ), enhance the accuracy of knowledge retrieval through contrastive learning. Assume that the model retrieves the most relevant positive sample knowledge entity k from the knowledge base and the user intention vector q + and several negative examples k - , using the contrast loss constraint similarity formula:

[0108]

[0109] Among them, sim is the cosine similarity, which calculates the similarity between vectors: τ is a temperature hyperparameter, usually 0.1, used to adjust the smoothness of the distribution, k + is a positive example knowledge entity that matches the user's intention (such as the product attributes of "red dress"), k - are negative knowledge entities that are irrelevant to the user’s intent (e.g., “blue shirt”).

[0110] Generate loss (L generate ) includes text fluency and image-text consistency optimization:

[0111] Text generation: Use cross entropy loss to constrain the generated response R and the true answer R gt Matches: Where T is the length of the generated text, P is the probability distribution of the word list output by the decoder, and R t is the tth word of the generated response, R ＜t is the first t-1 words to generate the response, X is the multimodal conversation history (text, image, etc.) of the input, and L text To optimize the fluency of generated text. Image-text matching: The semantic consistency between generated text and image feature v is constrained by triple loss. Given a positive image v + , generate text R and negative image v - , the loss function is: L image =max(0, sin(R, v - )-sin(R,v + )+α), where α is the interval threshold, ensuring that the similarity of positive examples is higher than that of negative examples, L image To ensure the semantic consistency between the generated text and the retrieved image. Ultimately, the generated loss is the weighted sum of the two: L generate =λ1L text +λ2L image , where λ1 and λ2 are balance weights, and the default value is 1.0.

[0112] The total loss function, the joint optimization objective is achieved by weighted fusion of three parts of loss:

[0113] L total =γ1L understand +γ2L reason +γ3L generate

[0114] y1, y2, and y3 are hyperparameters, defaulting to 1.0, that adjust the optimization intensity for each task. This model can synergistically improve intent understanding, knowledge retrieval, and multimodal generation capabilities. If intent classification is poor, increase γ1; if knowledge retrieval is inaccurate, increase γ2; and if generated responses are of low quality, increase γ3.

[0115] In a preferred embodiment, the multimodal data enhancement step is also included:

[0116] S61: Add noise perturbation to the speech data and convert it into text;

[0117] S62: Perform random cropping and color transformation on the image data to generate enhanced multimodal training samples.

[0118] In a preferred embodiment, the reply generation module supports dynamic output control:

[0119] S71: Selecting a single modal response of text, image, or voice based on the user intent classification result;

[0120] S72: In the recommendation scenario, a multimodal response is generated by jointly generating graphic descriptions and voice explanations.

[0121] The response generation module's dynamic output control mechanism flexibly selects the response format by analyzing user intent in real time. When a user makes a simple query, such as asking about the weather or navigation routes, the system prioritizes generating a single-modal response based on the intent classification results. If the user's intent is detected as "information confirmation" or "basic question and answer," the system only outputs a text response to ensure responsiveness. If the user is driving and requests route guidance, the system automatically switches to a voice response. When the user explicitly requests to view product images, the system directly returns the image content. This on-demand selection of a single modality reduces computational overhead and improves interaction efficiency.

[0122] In complex scenarios like e-commerce recommendations, the system activates a multimodal joint generation strategy. For example, if a user asks for "recommendations for red maxi dresses suitable for a beach vacation," the system first identifies the need for "outfit suggestions" through intent analysis. It then retrieves matching products from the knowledge base and simultaneously generates graphic cards showing close-ups of the dress's style and material, along with voice instructions explaining the sun protection factor and matching techniques. Graphical information helps users intuitively perceive the product's appearance, while voice supplements deepen their understanding of key selling points. This multimodal collaborative output significantly enhances the effectiveness of information communication. The dynamic control mechanism achieves a precise and efficient human-computer interaction experience through intent perception and scenario adaptation.

[0123] In a preferred embodiment, an incremental knowledge update mechanism is adopted: S81: real-time monitoring of changes in the external knowledge base, extraction of new entities and associations; S82: updating the key-value memory module through small sample fine-tuning to avoid model forgetting.

[0124] The incremental knowledge update mechanism ensures the model continuously adapts to new information through dynamic monitoring and local training. The system first scans the external knowledge base for changes in real time. For example, when new products are added or their attributes are modified on an e-commerce platform, it automatically identifies these newly added entities (such as "Sun Protection Index UPF50+ Dress") and their associated information (such as applicable scenarios and price). Upon detecting a change, the system extracts only the newly added data rather than reprocessing the entire knowledge base, thereby reducing computational overhead. Subsequently, a small-sample fine-tuning strategy is used to update the model's key-value memory module. For example, when the "Summer Sandals" category is added, the system trains the model using only a small amount of text-and-image conversation data containing this category, focusing on adjusting parameters related to the new entity (such as sandal material and matching suggestions), while retaining existing knowledge (such as the original clothing recommendation logic). This approach incorporates new knowledge while avoiding the problem of forgetting old knowledge caused by large-scale training, similar to adding a new dictionary entry without rewriting the entire book. This closed-loop monitoring-extraction-targeted update mechanism allows the model to absorb new knowledge while maintaining its original capabilities, significantly improving the system's adaptability and stability in dynamic scenarios.

[0125] In a preferred embodiment, the method is applied to an intelligent customer service system and specifically includes:

[0126] S91: Receive user voice input and convert it into text, and extract related image data;

[0127] S92: Call the multimodal understanding module to generate an intent vector and search the knowledge base to generate a response;

[0128] S93: Dynamically select text, voice, or mixed text and image formats to output a reply based on the conversation scenario.

[0129] The detailed process of calling the multimodal understanding module to generate an intent vector and searching the knowledge base to generate a response is as follows:

[0130] Input processing of the multimodal understanding module. The input type is the user's multimodal request (such as the text "Recommend a red dress suitable for the beach" + picture + voice supplement). Modal alignment includes text processing, image processing, and speech processing. Text processing: Extract text semantic features through models such as BERT and parse keywords (such as "beach", "red", "dress"). Image processing: Use Faster R-CNN to detect visual entities in the picture (such as "red chiffon skirt", "beach background") and extract regional features. Speech processing: Use ASR technology to convert speech into text, and then merge it with the original text for analysis (such as "high breathability requirements"). Feature fusion: Input text, image, and speech features into the cross-modal Transformer, and generate a unified joint representation through self-attention and cross-attention mechanisms.

[0131] Intent vector generation. Multi-round conversation context modeling. If the user previously asked about "summer outfits," the system encodes the conversation history through LSTM or Transformer to capture long-term dependencies. The current input is concatenated with the history to generate a context-aware semantic vector. Intent reasoning network: Uses a multi-hop attention mechanism to analyze user explicit needs (such as "recommended clothing") and implicit needs (such as "suitable for beach photos"). Output intent vectors to represent the user's core demands (e.g., vector dimensions are [scene: beach, type: dress, attributes: breathable and sun-protective]).

[0132] Knowledge base retrieval and reasoning. Knowledge base structure: Key: semantic vector of entity or intent (such as the embedding of "red chiffon skirt"). Value: associated multimodal data (such as image URL, price, applicable scenarios, user ratings). Retrieval process: Coarse-grained retrieval: calculate the cosine similarity between the intent vector and all keys in the knowledge base, and return the Top-K candidate entities (such as "red chiffon skirt" and "floral beach skirt"). Fine-grained reasoning: analyze the associated attributes of candidate entities through knowledge graphs (such as "chiffon corresponds to breathability" and "UPF50+ corresponds to sun protection"). Combined with user historical behavior (such as clicking on "cotton clothing"), filter out candidates that do not meet preferences. Dynamic knowledge update. If the knowledge base adds a new label "sun protection index UPF50+", update the key-value memory through small sample fine-tuning to avoid forgetting historical knowledge.

[0133] Multimodal response generation. Dynamic output control: Intent classification, select the response format based on the intent vector (such as the recommendation scenario is picture, text + voice). Text generation: Based on the Transformer decoder, extract attributes from the knowledge base to generate natural language (such as "Recommend XX brand red chiffon skirt, breathable and sun-proof"). Image matching: Retrieve the pictures most relevant to the text description (such as real photos of the beach), and ensure the consistency of pictures and texts through cross-modal comparison. Speech synthesis: Convert text responses into speech and add details (such as "sun protection index UPF50+"). Adversarial training: Introduce a discriminator to check the rationality of generation (such as avoiding "red skirt with blue shirt picture").

[0134] For example, a user inputs the text "Looking for a red long skirt suitable for the beach" and an accompanying image (a red chiffon skirt for a beach scene). Multimodal understanding: Text parsing extracts keywords such as "beach," "red," and "long skirt." Image parsing detects "chiffon material," "beach background," and "A-line skirt shape." Generates an intent vector: [Scene: Beach, Color: Red, Type: Long Skirt, Attributes: Breathable and Sun-Proof]. Knowledge base search: Matches Key = "red chiffon skirt," Value = [Image URL, Price ￥299, Scene: Beach, Sun Protection UPF50+]. The knowledge graph associates "Chiffon is associated with excellent breathability" and "Beach is associated with high sun protection requirements." Response generation: Text: "Recommend XX brand red chiffon skirt (with illustration), breathable and sun-proof, suitable for beach photos." Voice: "This long skirt has a sun protection factor of UPF50+ and is made of breathable chiffon fabric." Image card: Displays the product image and key attribute labels. This method achieves fine-grained alignment, precisely associating text keywords with image regions through cross-modal attention. Integrating the knowledge graph enhances logical reasoning (beach, sun protection). Top-K screening + knowledge filtering reduces computational overhead. Collaborative generation of text, images, and voice in a unified semantic space improves information density and user experience.

[0135] In practical applications of intelligent customer service systems, this method achieves efficient interaction through multimodal input processing and dynamic response generation. When a user initiates a request via voice, the system first converts the voice into text and simultaneously extracts user-uploaded or associated image data (such as images sent when describing a product issue). Next, the multimodal understanding module jointly analyzes the text and images, parsing the user's intent using a pre-trained model and generating an intent vector (for example, determining whether the user requires "product recommendations" or "after-sales consultation"). This intent vector is then used to retrieve information such as product attributes and user guides from the knowledge base. Finally, the system automatically selects the most appropriate response format based on the conversation scenario: simple queries prioritize text responses, complex operation instructions generate voice prompts, and product recommendations combine detailed text and images with voice explanations, such as displaying multi-angle images of a dress and supplementing the material and discount information with voice, thereby enhancing the naturalness of the interaction and the efficiency of information transmission.

[0136] A multimodal natural language understanding and generation system, including the following modules:

[0137] A pre-training module for training a multimodal encoder on image-text data through cross-modal associative learning to establish fine-grained modality alignment;

[0138] The fine-tuning module, which includes a hybrid prompt engine and a topic classifier, adapts to downstream tasks through trainable slots and injects human prior knowledge;

[0139] The intent reasoning module integrates a memory network and a multi-hop attention mechanism to achieve fine-grained retrieval and reasoning from user conversations to external knowledge;

[0140] The semantic fusion module uses a unified embedder to map text, images, and speech into the same semantic space and generate intent-aware query vectors;

[0141] The generation module outputs multimodal responses that match user intent based on a key-value memory library and adversarial training strategy.

[0142] This multimodal natural language understanding and generation system achieves intelligent conversation through a collaborative approach using five modules. Its core architecture inherits the pre-training-fine-tuning paradigm described in the method: First, fine-grained image-text alignment is established through cross-modal associative learning, followed by optimization of the model's capabilities for specific tasks. The pre-training module employs an associative mechanism. For example, when a user describes a "red dress," the system automatically associates regional features corresponding to the color and style in the product image. By exchanging text word vectors with image regional features, the model is forced to understand semantic equivalence, rather than relying on unimodal statistical patterns.

[0143] The fine-tuning module leverages experience in scenarios like e-commerce customer service to transform traditional question-and-answer tasks into fill-in-the-blank tasks. When a user asks, "What should I wear on a beach vacation?", the hybrid prompt engine rewrites the question into a statement: "Suitable beachwear should include [answer slot]." Combined with the topic classifier, the [topic slot] is filled with "sun-protective clothing," limiting candidate answers to categories like swimsuits and straw hats to prevent the model from incorrectly recommending down jackets due to data bias. The comparative example filling mechanism displays both positive examples like "breathable and quick-drying material" and negative examples like "fleece and thickening," improving answer accuracy through differential reinforcement learning.

[0144] The intent inference module uses a memory network to construct a user demand profile. For a multi-turn conversation like "Looking for a dress - want red - budget under 500 yuan," the multi-hop attention mechanism extracts consumer intent from keywords like "red" and "500 yuan," then leverages external knowledge bases to filter eligible products. The semantic fusion module maps diverse information, such as uploaded images of physical items and voice supplements like "want chiffon fabric," into a unified semantic space, ensuring a precise match between the lace pattern in the image and the text description.

[0145] The generation module selects the response format based on a dynamic routing strategy. Simple queries directly return text. Complex scenarios, such as clothing recommendations, utilize a key-value memory database. Using "red chiffon skirt" as the key, it associates values ​​such as price, applicable scenarios, and multi-angle images to generate a picture-text card with a voiceover explaining the sun protection factor. Adversarial training incorporates a discriminator to check image-text consistency, preventing the mismatch of a chiffon skirt with an image of jeans, ensuring logical consistency in multimodal content.

[0146] This invention uses cross-modal pre-training to enable the model to accurately match image and text details, and then fine-tunes to incorporate human experience. When a user asks a question, the system combines multiple rounds of dialogue and uploaded images to analyze intent, uses a memory network to retrieve relevant products from the knowledge base, and analyzes attribute associations through multi-hop attention. The system then maps text, images, and speech into a unified semantic space, ultimately dynamically generating graphic and text responses and synthesizing speech explanations. Adversarial training is used to avoid image and text mismatches, enabling precise and natural multimodal interaction.

[0147] In a preferred embodiment, the pre-training module further comprises:

[0148] Anchor detection unit, which extracts common entities from image-text pairs as alignment anchors;

[0149] Associative Mapping Unit, which optimizes modality alignment via mask reconstruction, contrastive loss, and a cross-modal associative classification (AMC) task;

[0150] The feature interaction unit uses a multi-layer Transformer to achieve cross-modal attention interaction.

[0151] The pre-training module achieves fine-grained semantic alignment between images and text through a multi-stage collaborative mechanism. First, the anchor detection unit extracts common entities from the input image-text pair as alignment benchmarks. For example, if an image shows a red dress and the text description contains the same keyword, the system automatically identifies this entity as an anchor for subsequent cross-modal alignment. This process is similar to focusing on key information when observing an image or reading text, ensuring that the model can locate core, relevant elements.

[0152] The associative mapping unit then optimizes modal alignment through three types of tasks. The mask reconstruction task randomly masks image regions or text words, requiring the model to predict the masked content based on the context, thereby learning cross-modal contextual dependencies. The contrastive loss task distinguishes between matching and mismatching image-text pairs, for example, narrowing the semantic distance between the "red dress" image-text pair while simultaneously distancing its association with unrelated image-text pairs. The cross-modal associative classification (AMC) task further strengthens entity-level alignment, for example, accurately associating the texture features of a skirt in an image with the "chiffon material" in the text to ensure consistency in the semantic space.

[0153] Finally, the feature interaction unit employs a multi-layer Transformer architecture to achieve deep cross-modal interaction. The model uses a self-attention mechanism to analyze the strength of associations between image regions and textual vocabulary, automatically capturing the implicit connection between a "seaside background" image and "vacation scene" text. It then fuses visual and linguistic features through multiple rounds of attention calculations, ultimately generating a unified semantic representation. This process combines the capabilities of visual observation and language understanding to provide robust cross-modal representation support for downstream tasks.

[0154] In a preferred embodiment, the fine-tuning module includes: a template generator that converts the question into a cloze template containing an [ANS] slot; an example filler that retrieves positive and negative examples from a knowledge base to fill the [S1]-[S3] slots; and a bias correction unit that generates answers based on visual features by constraining the model through contrastive loss.

[0155] The fine-tuning module guides the model to accurately answer visual questions through a hybrid prompt mechanism. First, the template generator converts user questions into cloze templates. For example, rewriting "What's suitable to wear at the beach?" into "A suitable outfit for the beach should include [answer]" aligns the question format with the mask reconstruction task from the pre-training phase, reducing task discrepancies. Next, the example filler retrieves positive and negative examples related to the question from the knowledge base. For example, for the question "sun protection clothing," it fills in the positive example "breathable and quick-drying material" and the negative example "fleece and thickened." This uses contrastive learning to strengthen the model's ability to distinguish correct answers. Finally, the bias correction unit introduces visual constraints. For example, when an image shows a green banana, even if the majority of the answer in the text data is "yellow," the model is forced to focus on the image features due to contrast loss, correcting for linguistic bias. This mechanism forces the model to reason based on a combination of image and text information, rather than relying on statistical patterns in the text, thereby improving the accuracy and interpretability of answers. In a preferred embodiment, the generation module is specifically implemented as: a knowledge query unit that retrieves entity-level knowledge from a key-value memory based on an intent vector; a multimodal decoder that supports the joint output of text generation, image retrieval, and speech synthesis; and a quality assessment unit that determines the rationality and diversity of responses through an adversarial network.

[0156] The generation module achieves accurate and diverse multimodal response output through multi-stage collaboration. The knowledge query unit first receives the intent vector delivered by the intent reasoning module. For example, if the user requests "recommend a red long skirt for a seaside vacation", the vector will carry semantic features such as scene, color, and category. The system performs similarity matching on the vector with the entity key in the key-value memory library, and selects the Top-K candidate entities (such as "red chiffon skirt" and "floral beach skirt"), and then combines the knowledge graph to analyze the entity attributes (such as "chiffon has excellent breathability" and "sun protection is required for seaside scenes"), and finally determines the best matching entity knowledge.

[0157] The multimodal decoder dynamically generates mixed responses based on the search results. The text generation module, based on the Transformer decoder, extracts entity attributes from the knowledge base, converting features like "chiffon material" and "UPF50+ sun protection" into natural language descriptions. The image retrieval module uses a cross-modal matching algorithm to select images from the product gallery that are most relevant to the text description. The speech synthesis module then feeds the generated text into a text-to-speech (TTS) model, generating a speech stream with additional details (e.g., "Sun protection index reaches UPF50+"). The three components are aligned in a unified semantic space to ensure consistency between text and image, and complementarity between speech and text.

[0158] The quality assessment unit optimizes the quality of the response through adversarial training. The generator is responsible for outputting multimodal responses, while the discriminator plays the role of a "quality inspector": on the one hand, it checks the semantic consistency of the image and text (such as avoiding "red skirt with blue shirt picture"), and on the other hand, it evaluates the diversity of the response (such as preventing repeated recommendations of the same product). When the discriminator identifies unreasonable content, the generator will adjust the parameters and regenerate until the output passes the verification. This mechanism forces the model to explore more diverse expressions while ensuring accuracy. For example, for the same dress, alternately display pictures from different angles or emphasize different selling points, thereby improving the user experience. In a preferred embodiment, the system is deployed on a cloud server and also includes: a real-time interface module that receives multimodal input data uploaded by the user terminal; a cache module that stores high-frequency knowledge query results to speed up the response; and an incremental update module that regularly synchronizes the external knowledge base and fine-tunes the model parameters.

[0159] This system is deployed on a cloud server and receives multimodal input data (such as text, images, and voice) uploaded by the user terminal through a real-time interface module, and distributes it to the multimodal understanding module for intent analysis. When a user initiates a request for "recommendation of red chiffon dresses", the interface module converts the voice into text, extracts the product image features, and passes the integrated multimodal data to the downstream module for processing. The cache module analyzes high-frequency query content (such as recent hot-selling styles and promotional information) based on user historical behavior, and stores the associated graphic features and price information in memory. For example, when a large number of users collectively query "summer sun protection clothing", the system automatically caches the image URLs, material descriptions, and applicable scenario labels of the relevant products, and subsequent similar requests can directly retrieve the results from the cache. The incremental update module monitors changes to the external knowledge base daily. If an e-commerce platform adds a new "UPF50+ Sun Protection Certification" label or adjusts product inventory, the module automatically captures the newly added entities and attributes and fine-tunes the entity vectors in the key-value memory using a small sample. For example, it encodes the newly added "Ice Silk Sun Protection Jacket" entity as a semantic vector and associates it with multimodal data. A rolling update strategy is also used to retain historical high-frequency data, ensuring that the model retains its original retrieval capabilities as it absorbs new knowledge. The three modules work together through an asynchronous communication mechanism, ensuring real-time response while continuously optimizing the system's accuracy and adaptability.

[0160] The present invention realizes intelligent interaction of multimodal dialogue through a multi-stage collaborative mechanism. The pre-training module establishes fine-grained semantic alignment of images and texts based on cross-modal associative learning, and forces the model to understand entity-level semantic equivalence through anchor detection and feature exchange, providing basic representation capabilities for downstream tasks; the fine-tuning module transforms the question-answering task into a cloze template with the help of a hybrid prompt mechanism, and incorporates human prior knowledge through dynamic topic classification and comparative sample filling to alleviate the language bias problem; the intention reasoning module combines the multi-hop attention mechanism with the memory network to extract intent vectors from multiple rounds of user dialogues, and links the knowledge base for coarse-grained retrieval and fine-grained reasoning; the semantic fusion module aligns text, image, and speech features through a unified embedding space, and uses image-text matching and word-region alignment tasks to strengthen global and local semantic associations; the generation module retrieves entity-level multimodal data through a key-value memory library based on intent-aware query vectors, and adopts an adversarial training strategy to dynamically generate collaborative responses of text, image, and speech. Through course learning, the system optimizes modal alignment, knowledge reasoning and generation modules in stages, combines cache acceleration and incremental update mechanisms, and realizes an efficient multimodal interaction closed loop in the cloud, significantly improving the accuracy of intent understanding and response generation in scenarios such as e-commerce customer service.

[0161] Although preferred embodiments of the present invention have been described, those skilled in the art may make additional changes and modifications to these embodiments once they become aware of the basic inventive concepts. Therefore, the appended claims are intended to be interpreted as covering the preferred embodiments and all changes and modifications that fall within the scope of the invention. The foregoing description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. It should be noted that any modifications, equivalent substitutions, and improvements made within the spirit and principles of the present invention are intended to be included within the scope of protection of the present invention.

Claims

1. A method based on multimodal natural language understanding and generation, characterized in that: Includes the following: S1: Construct a cross-modal pre-training module to train a multimodal encoder on an image-text dataset through self-supervised learning and establish a cross-modal associative mapping space; S2: Perform hybrid prompt fine-tuning to build cloze templates for downstream tasks, incorporating trainable prompt slots and human prior knowledge, and mitigate language bias through adaptive topic classification and comparative example filling; S3: Based on the intent reasoning network, combined with a memory mechanism and a hierarchical reply strategy, it extracts intent representations from multiple rounds of user conversations, retrieves external knowledge bases, and performs fine-grained reasoning; S4: Build a unified semantic representation framework to embed text, images, and speech into a unified space. Fine-tune through image-text matching and word-region alignment tasks to generate multimodal intent-aware query vectors. S5: A knowledge query module based on key-value memory generates entity-level multimodal responses and optimizes the semantic understanding and generation capabilities of the dialogue model through combinatorial fine-tuning.

2. The method according to claim 1, wherein: The cross-modal pre-training module described in S1 specifically includes: S11: Extract common entities of image-text pairs through anchor detection and construct anchor set; S12: A cross-modal associative learning strategy is adopted to model fine-grained semantic associations between modalities in the implicit mapping space through mask region reconstruction, cross-modal contrast loss, and associative map classification tasks. S13: Use the Transformer backbone network to perform multimodal feature interaction and output a joint representation that integrates vision and text.

3. The method according to claim 1, wherein: The hybrid prompt fine-tuning described in S2 includes: S21: Convert the question-answering task into a cloze template, rewrite the question into a declarative sentence using language reorganization rules, and insert trainable slots [ANS]; S22: Introducing a topic classification module to dynamically fill the [TOP] slot based on the user's historical conversations, narrowing the answer search scope; S23: Using contrastive learning strategy, retrieve positive and negative examples from the knowledge base to fill the [S1]-[S3] slots and generate candidate answer neighborhoods.

4. The method according to claim 1, wherein: The intent inference network described in S3 includes: S31: Extracting coarse-grained candidate knowledge from external knowledge base through memory network; S32: Use a multi-hop attention mechanism to analyze user intent and generate an intent vector as a query condition; S33: Combined with knowledge graph embedding, logical reasoning is performed on candidate knowledge and fine-grained reasoning paths are output.

5. The method according to claim 1, wherein: The implementation of the unified semantic representation framework described in S4 is as follows: S41: Use a multimodal embedder to encode text, images, and speech into vectors of the same dimension; S42: Optimizing global semantic alignment via image-text matching tasks; S43: Enhancing local entity-level associations through a word-region alignment task to generate user intent-aware query vectors.

6. The method according to claim 1, wherein: The combined fine-tuning strategy described in S5 includes: S51: Jointly optimize semantic understanding loss, knowledge reasoning loss, and generation loss; S52: A curriculum learning mechanism is used for three-stage training: first fine-tuning the modality alignment module, then optimizing the intent inference network, and finally jointly training the generation module. S53: Constraining generative results via adversarial discriminative networks.

7. The method according to claim 1, wherein: It also includes multimodal data enhancement steps: S61: Add noise perturbation to the speech data and convert it into text; S62: Perform random cropping and color transformation on the image data to generate enhanced multimodal training samples.

8. A multimodal natural language understanding and generation system, characterized in that: Includes the following modules: A pre-training module for training a multimodal encoder on image-text data through cross-modal associative learning to establish fine-grained modality alignment; The fine-tuning module, which includes a hybrid prompt engine and a topic classifier, adapts to downstream tasks through trainable slots and injects human prior knowledge; The intent reasoning module integrates a memory network and a multi-hop attention mechanism to achieve fine-grained retrieval and reasoning from user conversations to external knowledge; The semantic fusion module uses a unified embedder to map text, images, and speech into the same semantic space and generate intent-aware query vectors; The generation module outputs multimodal responses that match user intent based on a key-value memory library and adversarial training strategy.

9. A multimodal natural language understanding and generation system according to claim 8, characterized in that: The pre-training module further includes: Anchor detection unit, which extracts common entities from image-text pairs as alignment anchors; The associative mapping unit optimizes modal alignment through mask reconstruction, contrast loss and cross-modal associative classification tasks; the feature interaction unit uses a multi-layer Transformer to achieve cross-modal attention interaction.

10. A multimodal natural language understanding and generation system according to claim 8, characterized in that: The fine-tuning module includes: Template generator, which converts questions into cloze templates with [ANS] slots; The sample filler retrieves positive and negative samples from the knowledge base to fill the [S1]-[S3] slots; The bias correction unit constrains the model to rely on visual features to generate answers through contrastive loss.

Images