AI multimodal dialogue system based on multimodal recognition

Abstract

This application relates to the field of intelligent dialogue, and specifically discloses an AI multimodal dialogue system based on multimodal recognition, which first extracts text word granularity and image local features respectively, and then constructs a dynamic association between text and image at the feature level through a two-way cross-attention mechanism. Subsequently, the dynamic association of cross-modal high-level semantics is captured through multi-head attention. Its hierarchical processing mechanism enables the system to adapt to the value weights of multimodal information at different task stages, solving the modal gap problem caused by simple splicing of traditional methods, and capturing cross-modal spatiotemporal associations through a dynamic attention mechanism, and finally achieving accurate response through intent recognition and large model generation. In this way, it breaks through the limitations of existing technologies in cross-modal feature extraction and dynamic interaction modeling, and significantly improves the accuracy of multimodal intent understanding and response relevance.

Description

Technical Field

[0001] The present application relates to the field of intelligent dialogue, and more specifically, to an AI multimodal dialogue system based on multimodal recognition. Background Art

[0002] With the advancement of artificial intelligence, traditional text-based dialogue systems are facing limitations due to their inability to process multimodal information such as images and voice. Human communication is inherently multimodal, and combining visual and auditory information can enhance the naturalness and intelligence of interactions. Therefore, building dialogue systems that integrate multimodal information such as text and images has become a growing trend. By incorporating image information, the system can more accurately understand the specific object, scene, or emotion being referred to by the user, thereby providing more precise and personalized responses and significantly enhancing the human-computer interaction experience.

[0003] However, effectively fusing information from different modalities is a challenging problem. Different modalities exhibit significant differences in structure, features, and semantics (modal heterogeneity), making simple splicing ineffective fusion difficult. The correlations between multimodal information are complex and dynamic across time and space, and the value of each modality constantly changes at different stages or tasks. Modal interaction involves multiple levels, from low-level feature coupling to high-level semantic complementarity, requiring fusion strategies capable of handling both dynamics and hierarchical aspects. Existing multimodal dialogue systems employ relatively simple fusion methods for modal fusion, such as direct concatenation of feature vectors or simple attention mechanisms. These methods struggle to adequately address these challenges, resulting in poor fusion results and prone to biases in understanding user cross-modal intent, which in turn affects the accuracy and relevance of responses. In particular, in applications requiring a detailed understanding of the complex correspondence between text and images, existing technologies are insufficient in extracting and fusing cross-modal features of varying granularity and levels. This makes it difficult to capture the critical information necessary for accurate intent recognition, resulting in overall system performance failing to meet user demands for intelligent and natural interaction.

[0004] Therefore, how to design an efficient and robust multimodal information fusion strategy to overcome the gap between modalities, accurately capture the complex correlations between cross-modal information, and effectively use it for intent recognition and response generation is a key issue facing current AI multimodal dialogue systems. Summary of the Invention

[0005] In order to solve the above technical problems, this application is proposed.

[0006] According to one aspect of the present application, an AI multimodal dialogue system based on multimodal recognition is provided, which includes: an input acquisition module for acquiring text information input by a user and image data uploaded by a user; a multimodal encoding module for inputting the text information and the image data into a natural language unimodal depth encoder and an image depth encoder respectively to obtain a set of user intended word granularity semantic encoding vectors and a set of image local semantic encoding feature vectors; an early feature fusion module for inputting the set of user intended word granularity semantic encoding vectors and the set of image local semantic encoding feature vectors into an early feature fusion based on a cross attention network A component is used to obtain a set of enhanced user intention word granularity semantic coding vectors and a set of enhanced image local semantic coding feature vectors; a semantic layer feature fusion module is used to input the set of enhanced user intention word granularity semantic coding vectors and the set of enhanced image local semantic coding feature vectors into the semantic layer feature fusion component based on the multi-head attention module to obtain a multimodal fusion representation of user intention; an intention recognition module is used to input the multimodal fusion representation of user intention into the intention recognition classifier to obtain an intention recognition result; an intelligent response generation module is used to input the intention recognition result and the multimodal fusion representation of user intention into a large language model to obtain a response text.

[0007] Compared with the prior art, the present application provides an AI multimodal dialogue system based on multimodal recognition, which first extracts text word granularity and image local features respectively, laying a fine-grained foundation for cross-modal interaction. Secondly, through the bidirectional cross-attention mechanism, the dynamic association between text and image is constructed at the feature level, wherein the attention sparse processing screens out key cross-modal interaction nodes, and the residual enhancement module alleviates the feature mismatch problem caused by modal differences through a global-local symmetric optimization strategy, thereby strengthening the complementarity of low-level features. Subsequently, the dynamic association of cross-modal high-level semantics is captured through multi-head attention, and its hierarchical processing mechanism enables the system to adapt to the value weights of multimodal information at different task stages. This progressive fusion architecture from the feature layer to the semantic layer, combined with sparse coupling and symmetric optimization technology, not only solves the modal gap problem caused by the simple splicing of traditional methods, but also captures cross-modal spatiotemporal associations through a dynamic attention mechanism, and ultimately achieves accurate response through intent recognition and large model generation. In this way, the limitations of the prior art in cross-modal feature extraction and dynamic interaction modeling are broken through, and the accuracy of multimodal intent understanding and response relevance are significantly improved. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] The above and other purposes, features, and advantages of the present application will become more apparent through a more detailed description of the embodiments of the present application in conjunction with the accompanying drawings. The accompanying drawings are intended to provide a further understanding of the embodiments of the present application and constitute a part of the specification. Together with the embodiments of the present application, they are used to explain the present application and do not constitute a limitation of the present application. In the drawings, the same reference numerals generally represent the same components or steps.

[0009] Figure 1 4 is a block diagram of an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application.

[0010] Figure 2 4 is a block diagram of a multimodal encoding module in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application.

[0011] Figure 3 4 is a block diagram of an early feature fusion module in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application.

[0012] Figure 4 This is a block diagram of an image cross-modal feature fusion unit in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] The following describes embodiments of the present disclosure in more detail with reference to the accompanying drawings. While the drawings illustrate certain embodiments of the present disclosure, it should be understood that the present disclosure can be implemented in various forms and should not be construed as limited to the embodiments described herein. Rather, these embodiments are provided to provide a more thorough and complete understanding of the present disclosure. It should be understood that the drawings and embodiments of the present disclosure are for illustrative purposes only and are not intended to limit the scope of protection of the present disclosure.

[0014] It should be understood that the various steps described in the method embodiments of the present disclosure may be performed in a different order and / or in parallel. Furthermore, the method embodiments may include additional steps and / or omit the steps shown. The scope of the present disclosure is not limited in this respect.

[0015] This application is proposed to address the problems in the above background technology. Figure 1 : is a block diagram of an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application. Specifically, Figure 1As shown, according to an embodiment of the present application, the AI ​​multimodal dialogue system 100 based on multimodal recognition includes: an input acquisition module 110, which is used to acquire text information input by a user and image data uploaded by a user; a multimodal encoding module 120, which is used to input the text information and the image data into a natural language single-modal depth encoder and an image depth encoder respectively to obtain a set of user intended word granularity semantic encoding vectors and a set of image local semantic encoding feature vectors; an early feature fusion module 130, which is used to input the set of user intended word granularity semantic encoding vectors and the set of image local semantic encoding feature vectors into an early feature fusion group based on a cross-attention network. The present invention relates to a component for obtaining a set of enhanced user intention word granularity semantic coding vectors and a set of enhanced image local semantic coding feature vectors; a semantic layer feature fusion module 140 is used to input the set of enhanced user intention word granularity semantic coding vectors and the set of enhanced image local semantic coding feature vectors into the semantic layer feature fusion component based on the multi-head attention module to obtain a multimodal fusion representation of user intention; an intention recognition module 150 is used to input the multimodal fusion representation of user intention into the intention recognition classifier to obtain an intention recognition result; an intelligent response generation module 160 is used to input the intention recognition result and the multimodal fusion representation of user intention into a large language model to obtain a response text.

[0016] Specifically, the input acquisition module 110 is used to obtain text information input by the user and image data uploaded by the user. It should be understood that if only text information is relied upon, it is difficult to capture the user's intention, context or specific reference object expressed through non-verbal forms such as images. Given that human communication is inherently multimodal, users may use text and images simultaneously or alternately to express their needs when initiating a conversation (for example, what is the name of the plant in this picture?). Therefore, obtaining text information input by the user and image data uploaded by the user can capture all relevant information input by the user, laying the foundation for subsequent deep multimodal understanding, ensuring that the system can receive and integrate raw data streams from different modalities, thereby fully and accurately understanding the user's true intentions and situation, which is a prerequisite for achieving intelligent, natural and efficient multimodal conversations.

[0017] In particular, in one possible embodiment, the input acquisition module acquires information through the following method: This can be accomplished through the interactive interface provided by the system to the user. Specifically, the user enters their text message through a text input box. When the user completes their input and submits it (for example, by clicking the Send button or pressing the Enter key), the text information entered by the user is captured and transmitted as a raw string data stream to the system's input acquisition module. The interface also provides an image upload function, such as an upload image button or area. When the user clicks this function and selects an image file (for example, a .jpg or .png file) from their local device (such as a computer or mobile phone), the user's device reads the image file's raw binary data and sends it over the network to the server-side input acquisition module. The input acquisition module is responsible for monitoring and receiving these different data streams from the user interface, including text strings and image binary data. Upon receiving the data, the input acquisition module performs preliminary processing on the raw data, such as formatting checks, before converting it into a format suitable for further processing within the system and passing it to the subsequent multimodal encoding module.

[0018] Specifically, the multimodal encoding module 120 is configured to input the text information and the image data into a natural language unimodal deep encoder and an image deep encoder, respectively, to obtain a set of semantic encoding vectors at the granularity of user-intended words and a set of local semantic encoding feature vectors of images. Accordingly, as described in the background art, a key challenge in building a multimodal dialogue system lies in processing the modal differences and complex associations between different modal information such as text and images. Original text information (such as a character sequence) and image data (such as a pixel matrix) have essential differences in structure, feature space, and semantic expression, making them incapable of direct and effective comparison or fusion. Therefore, inputting the text information and the image data into a natural language unimodal deep encoder and an image deep encoder, respectively, can convert these original, heterogeneous modal data into their respective high-dimensional, semantically rich feature spaces, providing a unified, standardized semantic representation for subsequent multimodal fusion.

[0019] Specifically, in a specific example of this application, Figure 2 FIG is a block diagram of a multimodal encoding module in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application. Figure 2As shown, the multimodal encoding module 120 includes: a text segmentation processing unit 121, which is used to perform segmentation processing on the text information to obtain a text word sequence; a text semantic encoding unit 122, which is used to input the text word sequence into the natural language unimodal depth encoder to obtain a set of user intention word granularity semantic encoding vectors, wherein the natural language unimodal depth encoder is a semantic encoder including a Bert model; an image block processing unit 123, which is used to perform image block processing on the image data to obtain an image block sequence; an image semantic encoding unit 124, which is used to input the image block sequence into the image depth encoder to obtain a set of image local semantic encoding feature vectors, wherein the image depth encoder is a Vit model including an image block embedding coding layer.

[0020] It should be understood that text information, as an unstructured character sequence, is difficult to be directly understood and processed by deep learning models in its original form to extract deep semantic features. Natural language processing models, especially deep encoders based on the Transformer structure, are trained and inferred based on discrete vocabulary units or their subunits. Therefore, word segmentation processing of the text information to obtain a text word sequence can decompose the continuous text string input by the user into a series of basic units (i.e., words or tokens) with independent semantic meanings or constituting semantic units, forming an ordered sequence, thereby providing data support for the subsequent extraction of semantic encoding features of user intentions at the word granularity.

[0021] In particular, in one possible embodiment, the text segmentation unit 121 is implemented as follows: First, preprocessing is performed, such as removing noise from the text (e.g., standardizing special symbols and whitespace characters) and converting full-width to half-width characters. Then, the preprocessed text is segmented according to preset segmentation rules or dictionaries. Common segmentation methods include dictionary-based maximum matching, statistical model-based methods (e.g., hidden Markov models and conditional random fields), or more advanced methods based on deep learning models. For example, for Chinese text, since there are no obvious delimiters between words, the segmentation unit identifies word boundaries in the text based on a built-in vocabulary and language model, segmenting continuous Chinese character sequences into independent words. For example, if the input text message is: "What is the name of the plant in this picture?", the resulting text word sequence after segmentation may be ["this", "a", "picture", "in", "of", "plant", "called", "what", "?"]. The word segmentation method and dictionary used will be selected and set according to the specific language and application scenario, aiming to generate word sequences that can accurately reflect the semantic structure of the text and provide high-quality input for the subsequent semantic encoding step.

[0022] Correspondingly, the original text information is processed by word segmentation to obtain a discrete text word sequence. Although word segmentation divides continuous text into word units, these words themselves are still symbolic representations and cannot directly capture their deep semantics, contextual relationships, and subtle meanings in the expression of user intent. Different words may have different meanings in different contexts, and simple symbolic representations cannot reflect this contextual dependence. Therefore, the text word sequence is input into the natural language unimodal deep encoder to obtain a set of user intent word granularity semantic encoding vectors, which can convert these discrete word symbols into continuous, high-dimensional, and semantically rich vector representations. Through the natural language unimodal deep encoder, the system can learn and extract the deep semantic features of each word or token in the text in the entire sentence and even in a broader context, thereby obtaining a more expressive user intent word granularity semantic encoding vector, providing a basis for subsequent cross-modal fusion and intent recognition.

[0023] In particular, in a possible example, the text semantic encoding unit 122 is implemented as follows: for example, for the input text (What is the name of the plant in this picture?), after word segmentation, it can be converted into a word sequence ["this", "piece", "picture", "in", "of", "plant", "called", "what", "?"]. This word sequence is input into the natural language unimodal deep encoder, which includes a Bert model. The Bert model is a bidirectional encoder based on the Transformer architecture, and its core consists of a stack of multiple identical Transformer encoder layers. Before the word sequence is input into the Bert model, it is first labeled and mapped to a token sequence in the vocabulary preset by the Bert model. Each token is converted into its corresponding initial token embedding vector, and its position information in the sequence (position encoding) and possible paragraph information (paragraph encoding, if processing multiple paragraphs of text) are combined to form an input representation. These input representations are then passed through multiple Transformer encoder layers of the Bert model. Within each layer, using a multi-head self-attention mechanism, the model calculates the strength of association between each token and all other tokens in the sequence, thereby learning a contextualized representation of each token within its global context. These contextualized representations are then further nonlinearly transformed by a feedforward neural network. After processing through all Transformer layers, the Bert model outputs a sequence of vectors equal in length to the input token sequence, where each vector represents the final contextualized semantic encoding of the corresponding token. For example, for the input token sequence "this", "a", "picture", "in", "of", "plant", "called", "what", "?", the Bert model will output a set of nine vectors, one corresponding to the semantic encoding of a token. The resulting sequence of vectors is the set of semantic encoding vectors at the granularity of the user's intended word. These vectors capture the semantic and grammatical information of each word or token within the context of the entire text, providing a fine-grained representation of text features for subsequent modal fusion. The BERT model is pre-trained on a large-scale text corpus to learn general language understanding capabilities, and can then be fine-tuned on specific tasks (such as multimodal dialogue in this system) to better adapt to task requirements.

[0024] It should be understood that raw image data exists in the form of a pixel matrix. This two-dimensional spatial structure has significant modal differences from the linear sequence structure of text information. Directly using raw pixel data for subsequent cross-modal fusion and semantic understanding is difficult because the pixel-level information is too low-level to directly reflect the semantic content of objects, scenes, or local areas in the image. In order to process image data, it needs to be converted into a serialized form that can align with or interact with the text sequence to some extent. This allows the image data to be processed by the subsequent encoder and extract features with local semantic meaning.

[0025] In particular, in one possible embodiment, the image block processing unit 123 is implemented as follows: First, a preset image block size is determined, for example, the image block size is set to 16x16 pixels. This size is an important hyperparameter, and its setting depends on the specific model design, computing resources, and the requirements for local feature granularity, and is determined before model training. Next, the image block processing unit will divide the input original image into multiple small image blocks in the horizontal and vertical directions without overlapping according to this preset block size. For example, for a 224x224 pixel RGB image, if the image block size is set to 16x16 pixels, it will be divided into 224 / 16=14 blocks in the horizontal direction and 14 blocks in the vertical direction, resulting in a total of 14*14=196 image blocks. After the segmentation is completed, each image block is a pixel matrix of a fixed size (for example, 16x16x3, if the image is in RGB format). Finally, these two-dimensional image blocks are arranged in a specific order (for example, from left to right, top to bottom), and the pixels within each block are flattened into a one-dimensional vector (for example, a 16x16x3 = 768-dimensional vector), thus forming an ordered sequence of multiple image block vectors. This sequence of flattened image block vectors is the image block sequence. For example, for the 224x224 image above, after processing, the resulting sequence is a 196-length sequence, where each element is a 768-dimensional vector.

[0026] Accordingly, although the image is organized into a sequence of image blocks after block processing, each block is still a flattened representation of the original pixel data. This low-level pixel feature fails to capture high-level visual information such as the semantic content, texture, and shape of the local image region, nor can it reflect the spatial relationships and interactions between image blocks. In order for the system to understand the image content and effectively integrate it with textual information across modalities, it is necessary to convert these raw image blocks into feature vectors with rich semantic expression capabilities.

[0027] Specifically, in one possible embodiment, the image semantic encoding unit 124 is implemented as follows: for an image segmented into 196 16x16 pixel blocks, the input is a sequence of 196 flattened block vectors, each vector having, for example, 768 dimensions. The image semantic encoding unit inputs this block sequence into the image depth encoder, which includes a ViT model based on a block embedding layer. The ViT model first processes the input block sequence through a block embedding layer. This embedding layer is a linear projection layer that maps each flattened block vector (e.g., 768 dimensions) into a lower-dimensional embedding space (e.g., setting the embedding dimension to 768 or 1024 dimensions), resulting in a block embedding vector. This process converts the original pixel values ​​into a more abstract and compact feature representation. To preserve the spatial position information of the blocks (because the Transformer's self-attention mechanism is inherently position-independent), the ViT model adds a learnable or fixed positional encoding to the block embedding vector. These position-informed patch embeddings are then fed into the Transformer encoder portion of the ViT model, which consists of multiple stacked self-attention layers and a feed-forward neural network. Through the self-attention mechanism, the model computes the correlation between each patch embedding and the embeddings of all other patches in the sequence, thereby learning a contextualized representation of each local region within the context of the entire image. After processing through all Transformer layers, the ViT model outputs a sequence of 196 patch vectors, each of which represents the final contextualized semantic encoding of the corresponding patch. This vector sequence is the set of feature vectors representing the local semantic encodings of the image. For example, for an input sequence of 196 patches, the ViT model will output a set of 196 vectors, one for each patch's local semantic encoding. The ViT model is pre-trained on large-scale image datasets (such as ImageNet) to learn the ability to extract visual features, and can then be fine-tuned for the specifics of multimodal conversational tasks.

[0028] Specifically, the early feature fusion module 130 is configured to input the set of user-intended word granular semantic encoding vectors and the set of image local semantic encoding feature vectors into an early feature fusion component based on a cross-attention network to obtain a set of enhanced user-intended word granular semantic encoding vectors and a set of enhanced image local semantic encoding feature vectors. It should be understood that although text information and image data have been converted into their own sets of semantic encoding vectors through unimodal deep encoders, these vectors contain rich semantic information within their respective modalities, but they are still independently generated representations and lack direct interaction and association between modalities. User intent in multimodal conversations is often the result of the collaborative expression of text and image information. For example, text may refer to a specific object in an image, or the image provides context for the text. Simple splicing or independent semantic representations cannot capture this fine-grained correspondence and mutual influence across modalities. Therefore, it is necessary to design a sophisticated interaction mechanism at this relatively early feature level to enable fine-grained information exchange and mutual enhancement between features of different modalities.

[0029] Specifically, in a specific example of this application, Figure 3 FIG is a block diagram of an early feature fusion module in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application. Figure 3 As shown, the early feature fusion module 130 includes: a user intention semantic vector extraction unit 131, which is used to extract a first user intention word granularity semantic coding vector from the set of user intention word granularity semantic coding vectors; an image-oriented early cross-modal attention calculation unit 132, which is used to calculate the early feature cross-modal attention value of each image local semantic coding feature vector in the set of image local semantic coding feature vectors relative to the first user intention word granularity semantic coding vector to obtain a set of image-oriented early feature cross-modal attention values; an image cross-modal feature fusion unit 133, which is used to fuse the set of image local semantic coding feature vectors based on the set of image-oriented early feature cross-modal attention values ​​to obtain an image-oriented cross-modal early feature attention interaction coding vector; a user intention residual enhancement unit 134, which is used to input the image-oriented cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector into a residual module to obtain an enhanced user intention word granularity semantic coding vector corresponding to the first user intention word granularity semantic coding vector.

[0030] Accordingly, the text semantic encoding unit outputs a set of semantic encoding vectors at the user's intended word granularity. This set contains the deep-encoded vector representations of each word or token in the text sequence. In the subsequent early feature fusion stage, a cross-attention network-based approach is employed, where elements of one modality (e.g., text) are used as queries or key-value pairs to attend to elements of the other modality (e.g., image). In some cross-attention mechanism designs, a representative vector is extracted from the query modality as the overall query or starting point. This guides the attention process, focusing on the most relevant information in the query modality and interacting with the key-value pair modality. Therefore, a specific vector in the text modality is required as a starting point for computing early cross-modal attention between local semantic features of the text and image. This drives information flow from the image modality to the text modality (or vice versa, depending on the specific direction of the cross-attention), achieving early feature interaction and enhancement.

[0031] In particular, in one possible instance, the user intention semantic vector extraction unit 131 is implemented as follows: this set is an ordered vector sequence, corresponding to the token sequence obtained after the original text is segmented and tokenized. For example, if the original text (What is the name of the plant in this picture?) is processed to obtain a token sequence, and the natural language unimodal deep encoder (outputs a vector set corresponding to this sequence {vthis, vpictures, vinside, vof, vplant, vthing, vcalled, vwhat, v?}, where each vtoken is a high-dimensional (e.g., 768-dimensional) semantic encoding vector. The user intention semantic vector extraction unit selects and extracts a specific vector from this vector set as the first user intention word granularity semantic encoding vector. This extracted vector is then used to drive the early cross-modal attention calculation from image to text.

[0032] It should be understood that although text and images have been separately encoded as sets of semantic vectors, in the early fusion stage, it is necessary to establish intermodal associations so that features from one modality can perceive and utilize relevant information from the other. In particular, to better align and enhance local image features with the user's intent expressed in the text, the system needs to know which image regions are most relevant to the specific vector in the text that represents the user's core intent. Therefore, it is necessary to quantify the similarity or degree of association between each local region in the image (represented by its semantic encoding feature vector) and the representative semantic vector extracted from the text. This involves calculating the early feature cross-modal attention value of each local image semantic encoding feature vector in the set of local image semantic encoding feature vectors relative to the first user-intended word-granular semantic encoding vector. This similarity measure is converted into an attention weight, indicating the importance of each local image feature in responding to the specific intent of the text, thereby guiding the subsequent feature fusion process and achieving guided enhancement of image features by text.

[0033] In particular, in one possible embodiment, the image-to-early cross-modal attention calculation unit 132 is implemented as follows: the unit receives two inputs: the first user-intended word granularity semantic encoding vector provided by the user-intended semantic vector extraction unit (serving as a query in the attention mechanism, referred to as the query vector) and the set of image local semantic encoding feature vectors provided by the image semantic encoding unit (serving as a key in the attention mechanism, referred to as the key vector set). For example, the first user-intended word granularity semantic encoding vector is a high-dimensional vector representing text intent, with a dimension set to 768; the set of image local semantic encoding feature vectors includes multiple high-dimensional vectors representing local areas of the image, for example, 196 768-dimensional vectors.

[0034] The implementation is based on the scaled dot-product attention mechanism. First, the input query vector is projected into the query space via a learnable linear transformation (implemented by a weight matrix). Simultaneously, each image's local semantic encoding feature vector in the input key vector set is projected into the key space via another learnable linear transformation (implemented by a weight matrix). The parameters of these two linear transformation weight matrices are automatically learned during system training by optimizing the overall task objective. Their purpose is to map features from different modalities into a common, compatible representation space, enabling effective similarity comparisons.

[0035] Next, the dot product similarity between the projected query vector and each projected key vector is calculated to obtain the original attention score. For example, for a set of 196 image local feature vectors, 196 original attention scores will be obtained, each score corresponding to the similarity between a local image feature and the text query vector. In order to stabilize the calculation and avoid the dot product result being too large when the dimension is high, each original attention score is divided by the square root of the dimension of the projected key vector. Finally, the Softmax function is applied to all scaled scores. The Softmax function converts these scores into a set of probability values ​​so that the sum of the probability values ​​corresponding to all image blocks is 1. This set of probability values ​​output by the Softmax function is the set of cross-modal attention values ​​of the image to the early features. This set reflects the degree of attention or importance weight of the specific intention of the text to each local area of ​​the image.

[0036] Accordingly, the set of image local semantic encoding feature vectors contains the semantic information of each local area after the image is segmented, and the set of image-to-early feature cross-modal attention values ​​calculated in the previous step quantifies the association strength or importance between each image local feature and the first user intent word granularity semantic encoding vector. In multimodal interaction, the user's textual intent is often directed to a specific object or area in the image. Simply using the mean or concatenation of all local features of the image to represent the image may introduce a large amount of noise information that is irrelevant to the textual intent and fail to highlight visual elements that are closely related to the textual content. Therefore, it is necessary to use the calculated attention value as a weight to perform weighted aggregation on the set of image local semantic encoding feature vectors.

[0037] Specifically, in one possible implementation example of the present application, Figure 4 FIG is a block diagram of an image cross-modal feature fusion unit in an AI multimodal dialogue system based on multimodal recognition according to an embodiment of the present application. Figure 4 As shown, the image cross-modal feature fusion unit 133 includes: an attention weight value calculation subunit 1331, which is used to input the set of image-to-early feature cross-modal attention values ​​into a Softmax activation function to obtain a set of image-to-early feature cross-modal attention weight values; an attention weight value sparsification subunit 1332, which is used to input the set of image-to-early feature cross-modal attention weight values ​​into an attention sparsification module based on a preset threshold to obtain a sparse set of image-to-early feature cross-modal attention weight values; an image cross-modal feature weighting subunit 1333, which is used to use the sparse set of image-to-early feature cross-modal attention weight values ​​as a weight to calculate the weighted sum of the set of image local semantic coding feature vectors to obtain the image-to-cross-modal early feature attention interaction coding vector.

[0038] It should be understood that in the previous step, the similarity or correlation strength between each local region in the image and the specific vector representing the user intent in the text was calculated, and a set of cross-modal attention values ​​from the image to the early features was obtained. These values ​​reflect the original relevance score of each local image feature to the text intent. However, in order to effectively use these scores to fuse local image features in subsequent steps, these scores in an arbitrary range need to be converted into a form with clear physical meaning, namely weights, so that these weights can represent how much information each local image feature should contribute in the fusion process, and the sum of these contributions is normalized.

[0039] In particular, in one possible embodiment, the attention weight value calculation subunit 1331 is implemented as follows: this subunit receives as input the set of image-to-early feature cross-modal attention values ​​output by the previous step. A Softmax activation function is applied to this set of values. The Softmax function converts a set of arbitrary real-valued values ​​into a probability distribution. Specifically, for each value in the input set, its exponent is first calculated (the power of the natural constant e). Then, all these exponential calculation results are added together to obtain a sum. Finally, the exponential result of each value is divided by this sum. For example, if the input set is {s1, s2, ..., sn}, then in the output weight set {h1, h2, ..., hn}, each weight hi is calculated as follows: hi is equal to e raised to the power of si, then divided by the sum of e raised to the power of s1, e raised to the power of s2, and so on to e raised to the power of sn. After this calculation process, the output is the set of image-to-early feature cross-modal attention weight values. This set also contains the same number of values ​​as the input set (e.g., 196 weight values), each of which is a non-negative real number between 0 and 1, and the sum of all values ​​in the set is strictly equal to 1. These weight values ​​directly indicate the relative contribution of each local feature of the image driven by the text intent.

[0040] Accordingly, the set of image-to-early feature cross-modal attention weight values ​​obtained by the Softmax function assigns a weight to each local area of ​​the image. However, in actual multimodal scenarios, the user's text intent may only be highly correlated with a few key areas in the image, while most areas in the image may be background or information irrelevant to the intent. Even for non-relevant areas, Softmax may assign them a very small non-zero weight. Directly using all these non-zero weights for fusion, on the one hand, will introduce noise irrelevant to the intent and reduce the fusion efficiency; on the other hand, processing all local features consumes a lot of computing resources. Therefore, the present application inputs the set of image-to-early feature cross-modal attention weight values ​​into an attention sparse module based on a preset threshold to obtain a sparse set of image-to-early feature cross-modal attention weight values ​​to identify and retain the weights corresponding to those key image local features that are truly highly correlated with the text intent, while eliminating or suppressing the weights corresponding to those areas with lower weights that are considered irrelevant to the text intent.

[0041] In particular, in a possible example, the attention weight value sparsification subunit 1332 is configured to perform attention sparsification processing on the set of the image-to-early feature cross-modal attention weight values ​​according to the following formula to obtain a sparse set of the image-to-early feature cross-modal attention weight values, wherein the formula is: ;in, is each image-to-early feature cross-modal attention weight value in the set of image-to-early feature cross-modal attention weight values, is the preset threshold, It is the attention sparse processing, is the image-to-early feature cross-modal attention weight value of each image in the sparse set of image-to-early feature cross-modal attention weight values. In particular, the preset threshold here is a floating point number between 0 and 1, and its specific value is determined by optimizing the model performance during the training process. After processing, only those local areas of the image whose attention weight is greater than or equal to the preset threshold will have their corresponding weight values ​​retained (or keep their original value), while those with weights less than The area where the attention is not visible will be ignored and its weight will become zero, thus forming a sparse set of attention weights.

[0042] It should be understood that the set of image local semantic encoding feature vectors contains the semantic information of all local areas of the image, while the sparse set of image-to-early feature cross-modal attention weight values ​​obtained in the previous step clearly indicates which local areas in the image are highly correlated with the first user intent word granularity semantic encoding vector (representing text intent) and quantifies the relative importance of these related areas. In order to integrate these scattered local image information into a single fused representation that can also highlight the focus of text intent, it is necessary to use sparse attention weights as a guiding mechanism to perform biased aggregation of local image features. The resulting image-to-cross-modal early feature attention interaction encoding vector is a more compact, more information-focused fused representation that has been enhanced by text intent information, providing more targeted and robust basic features for subsequent cross-modal understanding or task execution.

[0043] In particular, in a possible instance, the user intention residual enhancement unit 134 includes: a sparse coupling symmetric optimization subunit, which is used to perform global-local sparse coupling symmetric optimization on the image-toward cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector to obtain the image-toward cross-modal early feature attention interaction optimized coding vector and the first user intention word granularity semantic optimized coding vector; a first user intention feature enhancement subunit, which is used to input the image-toward cross-modal early feature attention interaction optimized coding vector and the first user intention word granularity semantic optimized coding vector into a residual module to obtain an enhanced user intention word granularity semantic coding vector corresponding to the first user intention word granularity semantic coding vector.

[0044] It should be noted that the image-toward cross-modal early feature attention interaction coding vector is a global image semantic attention interaction feature representation that integrates the set of image local semantic coding feature vectors, while the first user intention word granularity semantic coding vector is a local text semantic coding feature representation. When the two are input into the residual module, the first user intention word granularity semantic coding vector will cause significant sparsification of the semantic space through interpolation. Although the attention sparsification module based on the preset threshold also performs attention-weighted sparsification on the set of image local semantic coding feature vectors, there is still a significant sparsity imbalance between the two, which affects the expression effect of the enhanced user intention word granularity semantic coding vector obtained by the residual module. For this reason, it is necessary to perform symmetrical optimization based on global-local sparse coupling on the image-toward cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector.

[0045] Based on this, in an implementable example of the present application, the sparsely coupled symmetric optimization subunit is used to perform global sparsity residual modeling on the image-toward cross-modal early feature attention interaction encoding vector and the first user intention word granularity semantic encoding vector to obtain an image-toward cross-modal early feature attention interaction encoding global vector and a first user intention word granularity semantic encoding global vector, that is: ;in, is the image-to-cross-modal early feature attention interaction encoding vector, is the first user intention word granularity semantic encoding vector, It is subtracted by position point, is the vector two norm, It is added by position point. To calculate the square of each eigenvalue in the vector, is the logarithmic function value with the natural constant e as the base, It is added by position point. For absolute value calculation, is the global vector encoding the image-to-cross-modal early feature attention interaction, is the global vector of the semantic encoding of the first user intention word granularity; that is, it is integrated in the respective semantic spaces based on the residual probability density representation so as to obtain the global characteristics of sparsity difference through spatial integral modeling.

[0046] Using the image-oriented cross-modal early feature attention interaction encoding vector and the first user intent word granular semantic encoding vector as the local space coordinate basis, the smoothness mutual constraint factor of the global sparsity residual is calculated to obtain the image-oriented cross-modal constraint factor and the first user intent constraint factor, namely: ;in, is the image-toward cross-modal constraint factor, It is the first user intention constraint factor.

[0047] Based on the image-toward cross-modal constraint factor and the first user intention constraint factor, the image-toward cross-modal early feature attention interaction encoding vector and the first user intention word granularity semantic encoding vector are symmetric transformed and optimized to obtain the image-toward cross-modal early feature attention interaction optimized encoding vector and the first user intention word granularity semantic optimized encoding vector, namely: ;in, is the image-to-cross-modal early feature attention interaction optimized encoding vector, This is the first user-intended word-granular semantically optimized encoding vector. This allows for residual coupling of global sparsity on local probability density, while ensuring the geometric characteristics of the feature flow pattern in the semantic space through smoothness constraints. This corrects the sparsity imbalance through semantic class space symmetry transformation, improving the expressiveness of the enhanced user-intended word-granular semantic encoding vector obtained through the residual module.

[0048] Accordingly, during cross-modal interaction, the user's textual intent often needs to be refined and verified through image information. Although the previous step has calculated the image local feature fusion vector based on textual intent (the image-to-cross-modal early feature attention interaction optimization encoding vector), this is still a representation focused on the image side, while the original textual intent vector (the first user intent word granularity semantic optimization encoding vector) may contain more abstract or global semantic information. In order for the final textual intent representation to fully integrate related visual evidence and thus become more accurate, specific, and robust, the interaction information on the image side needs to be used in reverse to enhance the original textual intent representation, thereby correcting and supplementing the original textual intent vector.

[0049] In particular, in one possible embodiment, the first user intent feature enhancement subunit is implemented as follows: First, the image-oriented cross-modal early feature attention interaction optimized encoding vector and the first user intent word-granular semantically optimized encoding vector are concatenated in the feature dimension to form a larger joint vector (e.g., 1536 dimensions after concatenation). This concatenated vector contains both the image information obtained after the early cross-modal interaction and the original text intent information. Next, the concatenated vector is input into a feedforward neural network. This feedforward neural network is the main processing path in the residual architecture and consists of one or more fully connected layers (linear transformations) and nonlinear activation functions (e.g., ReLU). Its parameters (weights and biases) are learned during training. The feedforward neural network learns a residual representation for the original text intent vector from the joint vector. The dimensionality of this residual representation is designed to be consistent with the dimensionality of the first user intent word-granular semantically optimized encoding vector (e.g., 768 dimensions). Finally, the residual vector output by the feedforward neural network is element-wise added to the original input first user intent word-granular semantically optimized encoding vector. This summation operation is the core of the residual connection. The resulting vector is the enhanced user intent word granularity semantic encoding vector corresponding to the first user intent word granularity semantic encoding vector. In this way, the original text intent vector is enhanced with relevant information from the image side, generating a more discriminative and context-aware fusion representation.

[0050] In particular, in another specific example of the present application, the early feature fusion module further includes: an image semantic vector extraction unit for extracting a first image local semantic coding feature vector from the set of the image local semantic coding feature vectors; a text-oriented early cross-modal attention calculation unit for calculating the early feature cross-modal attention value of each user-intended word granular semantic coding vector in the set of the user-intended word granular semantic coding vector relative to the first image local semantic coding feature vector to obtain a set of text-oriented early feature cross-modal attention values; a text cross-modal feature fusion unit for fusing the set of the user-intended word granular semantic coding vectors based on the set of text-oriented early feature cross-modal attention values ​​to obtain a text-oriented cross-modal early feature attention interaction coding vector; and an image semantic residual enhancement unit for inputting the text-oriented cross-modal early feature attention interaction coding vector and the first image local semantic coding feature vector into a residual module to obtain an enhanced image local semantic coding feature vector corresponding to the first image local semantic coding feature vector. It should be understood that although the original set of image local semantic coding feature vectors contains visual information of each area of ​​the image, this information is purely visual and is not directly integrated into the user's textual intent information. However, the user's intention is directed to a specific object or area in the image and is accurately expressed by the text. In order to enable the local representation of the image to perceive and reflect the visual content related to the user's textual intention, thereby improving the effectiveness of subsequent cross-modal fusion and understanding, the present application inputs the set of the user intention word granularity semantic coding vectors and the set of the image local semantic coding feature vectors into an early feature fusion component based on a cross-attention network and performs local image enhancement to introduce text information so that each local feature of the image can learn and absorb the textual semantic context related to it, thereby enhancing the expression of the image local semantic coding feature vector. In particular, the specific implementation process of these three units is the same as the processing process of obtaining the enhanced user intention word granularity semantic coding vector by the user intention semantic vector extraction unit, the image to early cross-modal attention calculation unit, the image cross-modal feature fusion unit and the user intention residual enhancement unit in the early feature fusion module.

[0051] Specifically, the semantic layer feature fusion module 140 is configured to input the set of enhanced user intent word-granular semantic encoding vectors and the set of enhanced image local semantic encoding feature vectors into a semantic layer feature fusion component based on a multi-head attention module to obtain a multimodal fusion representation of user intent. Accordingly, after processing by the early feature fusion component based on a cross-attention network, a set of enhanced user intent word-granular semantic encoding vectors and a set of enhanced image local semantic encoding feature vectors that incorporate partial cross-modal information are obtained. Although the vectors in these sets already contain enhanced information from another modality, they are still discrete representations corresponding to the word granularity of the text or the local region of the image. In order to ultimately obtain a single multimodal understanding representation that can comprehensively and comprehensively represent the user's overall intent and image-related content, it is necessary to effectively aggregate and fuse these enhanced feature information distributed across different modalities and different local regions. This ensures that the resulting multimodal fusion representation of user intent will be a product of deep fusion of text and image information, directly serving subsequent intent understanding or task execution.

[0052] Specifically, in one possible embodiment, the semantic layer feature fusion module 140 is implemented as follows: First, the set of enhanced user intent word granular semantic encoding vectors and the set of enhanced image local semantic encoding feature vectors are concatenated into a unified vector sequence. Specifically, all M text vectors and all N image vectors are arranged in a certain order to form a sequence with a total length of M+N. This concatenated sequence serves as the input to the multi-head attention module. The multi-head attention module comprises several parallel attention heads. For example, the number of heads can be preset to 8, which is a hyperparameter determined through experimental optimization. Within each attention head, the input vector sequence is first mapped into a sequence of query, key, and value vectors through different linear transformations (i.e., multiplication with different learnable weight matrices and addition of biases). The parameters of these linear transformations are learned by the model during training. Next, the matrix product of the transposed query and key is calculated to obtain an attention score. The attention score is then scaled (divided by the square root of the key vector dimension) and a normalized exponential function (Softmax) is applied to obtain the attention weight. Finally, the attention weight matrix is ​​multiplied by the sequence of value vectors to obtain the output sequence of the attention head. This output sequence is also of length M+N, with each vector having a dimension of D divided by the number of heads (if divisible, or resized by a subsequent linear layer), for example, 768 / 8 = 96 dimensions. The output sequences of all attention heads are concatenated along the last dimension to form a higher-dimensional sequence (for example, 8*96 = 768 dimensions). This concatenated sequence then undergoes a final linear transformation to remap its dimensions back to the original vector dimension D (for example, 768 dimensions), resulting in the total output sequence of the multi-head attention module, which is still of length M+N and dimension D. To obtain a single multimodal fused representation of user intent from this sequence of M+N vectors, an aggregation operation is required. A commonly used approach is global average pooling, which computes the average of all M+N vectors in the multi-head attention module output sequence along each dimension. For example, the first dimension of all vectors is averaged to obtain the first dimension of the result vector; the second dimension of all vectors is averaged to obtain the second dimension of the result vector, and so on, until all D dimensions are calculated. The final single vector obtained in this way, namely the user intent multimodal fusion representation vector, is the user intent multimodal fusion representation. It integrates all enhanced text word granularity information and image local feature information, and captures the complex interactive relationship between them through the multi-head attention mechanism.

[0053] Specifically, the intent recognition module 150 is configured to input the multimodal fused representation of user intent into an intent recognition classifier to obtain an intent recognition result. It should be understood that the previous steps have deeply integrated the user's textual intent with key image information through a multi-level cross-modal attention mechanism, ultimately generating a compact and information-rich single vector, namely, the multimodal fused representation of user intent. This vector comprehensively reflects the meaning of the user's intent within the context of the image. In order for the system to make specific judgments or take actions based on this fused representation, this high-dimensional numerical representation needs to be converted into a discrete, understandable intent category. The classifier's role is to convert complex numerical features into clear semantic labels, thereby explicitly identifying the ultimate purpose or need expressed by the user input (text + image), enabling the system to understand the user's true intent and take appropriate subsequent processing steps. Specifically, the intent recognition classifier can be a feedforward neural network, such as a multilayer perceptron containing one or more hidden layers. Nonlinear activation functions are used between the hidden layers, and the output layer is a linear layer.

[0054] Specifically, in one possible embodiment, the intent recognition module 150 is implemented as follows: a multimodal fusion representation vector of user intent (multimodal fusion representation of user intent) is input into an intent recognition classifier. For example, the classifier is a feed-forward network with one hidden layer. The input vector first passes through the first linear transformation layer. In this layer, the input vector is matrix multiplied by a learnable weight matrix and then added with a learnable bias vector to produce an intermediate vector. For example, if the hidden layer is set to 512 dimensions, the weight matrix has dimensions of (512 × 768), and the bias vector has dimensions of 512. Next, a nonlinear activation function, such as a rectified linear unit (ReLU), is applied to this intermediate vector to introduce nonlinearity. The activated vector is then input into the output linear transformation layer. In this layer, the vector is again multiplied by a learnable weight matrix and added with a learnable bias vector. The weight matrix of the output layer has dimensions equal to (the number of intent categories + 512), and the bias vector has dimensions equal to the number of intent categories. The number of intent categories is equal to the total number of user intents that the system pre-defines for recognition, for example, 100 different intent categories. The output of this linear layer is a vector with a dimension equal to the number of intent categories, with each element representing the model's raw prediction score for the corresponding intent category. Next, a normalized exponential function (Softmax) is applied to this prediction score vector. The Softmax function converts the scores into a probability distribution, where each element ranges from 0 to 1 and the sum of all elements is 1. Each probability value indicates the likelihood that the model predicts the user input belongs to the corresponding intent category. The specific content of the intent recognition result depends on the system design; the most basic component is the identified intent category label. It also includes the confidence score corresponding to the predicted label (i.e., the highest probability value output by the Softmax function). For example, if the user inputs the text "What is the name of the plant in this picture?" and provides an image of a specific plant, the previous cross-modal fusion step generates a multimodal fused representation of the user intent, including both the textual "plant" and the plant features in the image. When this fused vector is input into the intent recognition classifier, the classifier calculates scores for all pre-defined intent categories. For example, the system's pre-defined intent categories include plant identification, product search, and navigation. After Softmax, the model might output probabilities for each category, such as 0.98 for plant identification, 0.01 for product search, and 0.005 for navigation. In this case, the intent recognition result is labeled "Plant Identification" and includes a confidence score of 0.98. All learnable parameters of the classifier (weight matrix and bias vector) are automatically determined during the model training phase by learning from a large number of multimodal user input samples labeled with intent categories (e.g., plant identification).

[0055] Specifically, the intelligent response generation module 160 is configured to input the intent recognition results and the multimodal fusion representation of user intent into a large language model to generate a response text. In particular, the previous steps have successfully fused the user's textual intent with relevant visual information in the image through complex cross-modal interaction and feature fusion into a single, high-dimensional multimodal fusion representation of user intent, and clarified the user's primary intent category through the intent recognition classifier. However, these internal representations and classification results are machine-understandable and insufficient for direct, natural, and humanistic communication with the user. Users expect a response text expressed in natural language that accurately and fluently addresses the intent and content implied by their multimodal input. Therefore, it is necessary to input the intent recognition results and the multimodal fusion representation of user intent into a large language model to generate a response text, converting it into a text output that conforms to human language habits. It is worth noting that the large language model, with its powerful text generation capabilities and ability to understand complex contexts, can comprehensively consider the user's specific intent, relevant details in the image, and the deeper meaning of their fusion, thereby generating a coherent, relevant, and high-quality response text, enabling effective interaction between the system and the user.

[0056] In particular, in a possible embodiment, the intelligent response generation module 160 is implemented as follows: First, the intent recognition result and the multimodal fusion representation of the user intent need to be converted into an input sequence form that can be processed by the large language model. The intent recognition result, such as the recognized intent label identifying plants, will be converted into a series of input tokens predefined by the model, such as a special token representing the intent type, or a text string describing the intent (such as the user wants to identify the plant in the picture). These tokens will be mapped into corresponding embedding vector sequences through the word embedding layer of the large language model. The multimodal fusion representation vector of the user intent, as a continuous high-dimensional vector, needs to be processed through a dedicated linear projection layer. This linear projection layer maps the 768-dimensional fusion vector to an embedding vector dimension that matches the internal representation dimension of the large language model (for example, if the hidden state dimension of the large language model is 1024 dimensions, it is mapped to a 1024-dimensional vector through a 768x1024 weight matrix and bias). Next, these transformed and projected embedding vector sequences (including embeddings from intent recognition results, embeddings from the multimodal fusion representation, and possibly embeddings from the user's original text) are concatenated in a predetermined order to form a complete input sequence. This sequence is fed into the core structure of the large language model, namely, the multi-layered Transformer modules. Within these layers, through self-attention mechanisms and feedforward networks, the model fully processes all information in the input sequence, understanding the meaning of intent labels and the contextual information carried by the multimodal fusion representation, and establishing complex relationships between them. The output sequence from the Transformer layers contains the model's deep understanding of the input context. Based on the representations processed by the Transformer layers, the large language model enters the text generation phase. Using an autoregressive approach, the model predicts the next token one by one based on the input context and the currently generated token sequence. This process begins with a starting token (e.g., a special token indicating the start of generation) and continues until a finishing token is generated or a preset maximum length is reached. The next word is predicted by passing the output of the last Transformer layer through a linear layer and a Softmax function. This yields a probability distribution for each word in the vocabulary as the next word. The model then selects the word with the highest probability, or uses a sampling strategy. The generated response text decodes this predicted word sequence back into natural language.

[0057] For example, if the user inputs the text "What is the name of the plant in this picture?" and provides a picture containing a specific plant, the intent recognition classifier outputs the intent recognition result of identifying the plant, and the multimodal fusion representation vector of the user's intent contains the fusion information of the text and the plant in the image. The intent to identify the plant is converted into an embedding, the fusion vector is projected into an embedding, and they are concatenated and input into the large language model. The large language model processes this information and understands that the user wants to know the name of the plant in the picture. Based on its training knowledge and the fusion features of the input, the model generates a response text, for example: Based on the image information, this plant is likely a tulip. The response text contains a response to the user's intent (such as "based on the image information") and the specific information identified based on the multimodal fusion results (such as "tulip"). The response text is intended to provide the information required by the user in natural language.

[0058] It's worth noting that all weight matrices and parameters used in the various components described in this application, including the natural language unimodal deep encoder (including the BERT model), image deep encoder (including the ViT model), early feature fusion component (based on the cross-attention module), semantic feature fusion component (based on the multi-head self-attention module), intent recognition classifier (feedforward neural network or multi-layer perceptron), and large language model, are not preset fixed values. These parameters are automatically acquired during the model training process by learning from a large number of labeled datasets. The training process typically involves performing forward computations on the training data to obtain outputs. The difference between the model outputs and the true targets (such as the correct intent label, expected response text, or matched features) is then calculated, measured by a loss function. Next, an optimization algorithm (such as gradient descent and its variants) is used to iteratively update the model's internal weight matrices and parameters based on the gradient of the loss function. Through repeated forward computations and parameter updates, the model gradually adjusts its internal parameters to minimize the loss function, thereby learning how to effectively encode, fuse, classify, and generate information.

[0059] In summary, the AI ​​multimodal dialogue system 100 based on multimodal recognition according to the embodiment of the present application is explained, which first extracts the text word granularity and the local features of the image respectively, laying a fine-grained foundation for cross-modal interaction. Secondly, through the two-way cross-attention mechanism, the dynamic association between text and image is constructed at the feature level, wherein the attention sparse processing screens out the key cross-modal interaction nodes, and the residual enhancement module alleviates the feature mismatch problem caused by modal differences through the global-local symmetric optimization strategy, and strengthens the complementarity of low-level features. Subsequently, the dynamic association of cross-modal high-level semantics is captured through multi-head attention, and its hierarchical processing mechanism enables the system to adapt to the value weights of multimodal information at different task stages. This progressive fusion architecture from the feature layer to the semantic layer, combined with sparse coupling and symmetric optimization technology, not only solves the modal gap problem caused by the simple splicing of traditional methods, but also captures cross-modal spatiotemporal associations through the dynamic attention mechanism, and finally achieves accurate response through intent recognition and large model generation. In this way, the limitations of existing technologies in cross-modal feature extraction and dynamic interaction modeling are overcome, and the accuracy and response relevance of multimodal intent understanding are significantly improved.

[0060] As described above, the AI ​​multimodal dialogue system 100 based on multimodal recognition according to the embodiment of the present application can be implemented in various wireless terminals, such as a server having an AI multimodal dialogue algorithm based on multimodal recognition. In one possible implementation, the AI ​​multimodal dialogue system 100 based on multimodal recognition according to the embodiment of the present application can be integrated into the wireless terminal as a software module and / or hardware module. For example, the AI ​​multimodal dialogue system 100 based on multimodal recognition can be a software module in the operating system of the wireless terminal, or it can be an application developed for the wireless terminal; of course, the AI ​​multimodal dialogue system 100 based on multimodal recognition can also be one of the many hardware modules of the wireless terminal.

[0061] Alternatively, in another example, the AI ​​multimodal dialogue system 100 based on multimodal recognition and the wireless terminal may also be separate devices, and the AI ​​multimodal dialogue system 100 based on multimodal recognition may be connected to the wireless terminal via a wired and / or wireless network and transmit interaction information in accordance with an agreed data format.

[0062] While various implementations of the present disclosure have been described above, the above description is intended to be illustrative and not exhaustive. The present disclosure is not limited to the disclosed implementations, and numerous modifications and variations will be apparent to those skilled in the art without departing from the scope and spirit of the described implementations.

Claims

1. An AI multimodal dialogue system based on multimodal recognition, characterized in that: include: An input acquisition module, used to acquire text information input by the user and image data uploaded by the user; A multimodal encoding module is used to input text information and image data into a natural language unimodal deep encoder and an image deep encoder respectively to obtain a set of user intended word granular semantic encoding vectors and a set of image local semantic encoding feature vectors; An early feature fusion module, configured to input the set of user intended word granular semantic encoding vectors and the set of image local semantic encoding feature vectors into an early feature fusion component based on a cross-attention network to obtain a set of enhanced user intended word granular semantic encoding vectors and a set of enhanced image local semantic encoding feature vectors; The semantic layer feature fusion module is used to input the set of enhanced user intent word granular semantic coding vectors and the set of enhanced image local semantic coding feature vectors into the semantic layer feature fusion component based on the multi-head attention module to obtain a multimodal fusion representation of user intent; The intent recognition module is used to input the multimodal fusion representation of user intent into the intent recognition classifier to obtain the intent recognition result; The intelligent response generation module is used to input the intent recognition results and the multimodal fusion representation of user intent into the large language model to obtain the response text; Among them, the early feature fusion module is further used to: Extracting a first user intention word granularity semantic encoding vector from a set of user intention word granularity semantic encoding vectors; Calculating the early feature cross-modal attention value of each image local semantic encoding feature vector relative to the first user intention word granularity semantic encoding vector to obtain a set of image-to-early feature cross-modal attention values; Based on the set of image-toward early feature cross-modal attention values, a set of image local semantic encoding feature vectors is fused to obtain an image-toward cross-modal early feature attention interaction encoding vector; Performing a global-local sparse coupling symmetric optimization on the image-toward cross-modal early feature attention interaction encoding vector and the first user intent word granular semantic encoding vector to obtain an image-toward cross-modal early feature attention interaction optimized encoding vector and a first user intent word granular semantic optimized encoding vector, wherein the global-local sparse coupling symmetric optimization is performed based on global sparsity residual modeling and smoothness mutual constraint of the global sparsity residual on the image-toward cross-modal early feature attention interaction encoding vector and the first user intent word granular semantic encoding vector; The image is input into the residual module into the cross-modal early feature attention interaction optimized coding vector and the first user intention word granularity semantic optimized coding vector to obtain the enhanced user intention word granularity semantic coding vector corresponding to the first user intention word granularity semantic coding vector.

2. The AI ​​multimodal dialogue system based on multimodal recognition according to claim 1, characterized in that: The multimodal encoding module includes: a text segmentation processing unit, which is used to perform segmentation processing on the text information to obtain a text word sequence; a text semantic encoding unit, which is used to input the text word sequence into the natural language unimodal deep encoder to obtain a set of user intended word granularity semantic encoding vectors, wherein the natural language unimodal deep encoder is a semantic encoder including a Bert model; an image block processing unit, which is used to perform image block processing on the image data to obtain an image block sequence; an image semantic encoding unit, which is used to input the image block sequence into the image depth encoder to obtain a set of image local semantic encoding feature vectors, wherein the image depth encoder is a Vit model including an image block embedding encoding layer.

3. The AI ​​multimodal dialogue system based on multimodal recognition according to claim 2, characterized in that: Based on the set of image-to-early feature cross-modal attention values, a set of image local semantic coding feature vectors is fused to obtain an image-toward cross-modal early feature attention interaction coding vector, including: inputting the set of image-to-early feature cross-modal attention values ​​into a Softmax activation function to obtain a set of image-to-early feature cross-modal attention weight values; inputting the set of image-to-early feature cross-modal attention weight values ​​into an attention sparse module based on a preset threshold to obtain a sparse set of image-to-early feature cross-modal attention weight values; using the sparse set of image-to-early feature cross-modal attention weight values ​​as a weight, calculating the weighted sum of the set of image local semantic coding feature vectors to obtain the image-toward cross-modal early feature attention interaction coding vector.

4. The AI ​​multimodal dialogue system based on multimodal recognition according to claim 3, characterized in that: Inputting the set of image-to-early feature cross-modal attention weight values ​​into an attention sparsification module based on a preset threshold to obtain a sparse set of image-to-early feature cross-modal attention weight values, comprising: performing attention sparsification processing on the set of image-to-early feature cross-modal attention weight values ​​according to the following formula to obtain a sparse set of image-to-early feature cross-modal attention weight values, wherein the formula is: ;in, is each image-to-early feature cross-modal attention weight value in the set of image-to-early feature cross-modal attention weight values, is the preset threshold, It is the attention sparse processing, It is the individual image-to-early feature cross-modal attention weight values ​​in the sparse set of image-to-early feature cross-modal attention weight values.

5. The AI ​​multimodal dialogue system based on multimodal recognition according to claim 4 is characterized in that: The image-toward cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector are symmetrically optimized based on global-local sparse coupling to obtain the image-toward cross-modal early feature attention interaction optimized coding vector and the first user intention word granularity semantic optimized coding vector, including: performing global sparsity residual modeling on the image-toward cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector to obtain the image-toward cross-modal early feature attention interaction coding global vector and the first user intention word granularity semantic coding global vector; using the image-toward cross-modal early feature The feature attention interaction coding vector and the first user intention word granularity semantic coding vector are used as the local space coordinate basis to calculate the smoothness mutual constraint factor of the global sparsity residual to obtain the image-oriented cross-modal constraint factor and the first user intention constraint factor; based on the image-oriented cross-modal constraint factor and the first user intention constraint factor, the image-oriented cross-modal early feature attention interaction coding vector and the first user intention word granularity semantic coding vector are symmetric transformed and optimized to obtain the image-oriented cross-modal early feature attention interaction optimized coding vector and the first user intention word granularity semantic optimized coding vector.

6. The AI ​​multimodal dialogue system based on multimodal recognition according to claim 1, characterized in that: The early feature fusion module also includes: an image semantic vector extraction unit, used to extract a first image local semantic coding feature vector from the set of image local semantic coding feature vectors; a text-oriented early cross-modal attention calculation unit, used to calculate the early feature cross-modal attention value of each user intention word granularity semantic coding vector in the set of user intention word granularity semantic coding vectors relative to the first image local semantic coding feature vector to obtain a set of text-oriented early feature cross-modal attention values; a text cross-modal feature fusion unit, used to fuse the set of user intention word granularity semantic coding vectors based on the set of text-oriented early feature cross-modal attention values ​​to obtain a text-oriented cross-modal early feature attention interaction coding vector; an image semantic residual enhancement unit, used to input the text-oriented cross-modal early feature attention interaction coding vector and the first image local semantic coding feature vector into a residual module to obtain an enhanced image local semantic coding feature vector corresponding to the first image local semantic coding feature vector.

Images