AI dialogue system based on multi-modal input

Abstract

The invention provides an AI dialogue system based on multi-modal input, relates to the field of AI intelligent interaction, and adopts a data analysis technology based on artificial intelligence to perform image semantic feature extraction on an intention expression image input by a user and perform text semantic understanding on an intention expression text input by the user. And intelligently generating a reply text based on features and cue words obtained by performing principal component analysis interactive fusion on the semantic coding features of the intention expression image and the semantic coding features of the intention expression text. Therefore, by analyzing the multi-modal intention expression data, the AI dialogue system can more accurately understand the actual intention of the user, so that more accurate answering is provided, and the user experience is improved.

Description

Technical Field

[0001] The present application relates to the field of AI intelligent interaction, and more specifically, to an AI dialogue system based on multimodal input. Background Art

[0002] With the rapid development of artificial intelligence technology, natural language processing technology has made great progress. This enables computers to understand and process human language more naturally, thus promoting the emergence and development of AI dialogue systems. AI dialogue systems use artificial intelligence technology to achieve human-computer dialogue. They can not only understand human language, but also simulate human dialogue methods, providing users with a more intelligent and convenient interactive experience.

[0003] Traditional AI dialogue systems usually rely solely on text input to understand and respond to user intent. This single-modality understanding method may not fully capture the complexity and nuances of user expressions, which may cause the system to misunderstand the user's actual intent, which in turn may lead to the system providing inaccurate responses, which can easily lead to a poor user experience.

[0004] Therefore, we look forward to an AI dialogue system based on multimodal input. Summary of the invention

[0005] In order to solve the above technical problems, this application is proposed. The embodiment of this application provides an AI dialogue system based on multimodal input.

[0006] According to one aspect of the present application, an AI dialogue system based on multimodal input is provided, which includes: a user input data acquisition module, used to acquire first modal data and second modal data input by a user, wherein the first modal data is an intention expression image, and the second modal data is an intention expression text; an intention expression related feature extraction module, used to perform data analysis on the first modal data and the second modal data respectively to obtain semantic coding features of the intention expression image and semantic coding features of the intention expression text; an intention expression image text feature joint encoding module, used to perform multimodal principal component compensation interactive joint encoding on the semantic coding features of the intention expression image and the semantic coding features of the intention expression text to obtain intention expression image-text joint features; a reply text generation module, used to generate a reply text based on the intention expression image-text joint features and prompt words.

[0007] Compared with the prior art, the AI ​​dialogue system based on multimodal input provided by the present application uses artificial intelligence-based data analysis technology to extract image semantic features of the intention expression image input by the user and to understand the text semantics of the intention expression text input by the user, and intelligently generates reply text based on the features and prompt words obtained by interactive fusion of the semantic coding features of the intention expression image and the semantic coding features of the intention expression text. In this way, by analyzing the multimodal intention expression data, the AI ​​dialogue system can more accurately understand the actual intention of the user, and then provide more accurate answers, which is conducive to improving the user experience. BRIEF DESCRIPTION OF THE DRAWINGS

[0008] In order to more clearly illustrate the embodiments of the present application or the technical solutions in the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0009] In the attached picture: Figure 1 4 is a system block diagram of an AI dialogue system based on multimodal input according to an embodiment of the present application.

[0010] Figure 2 Schematic diagram of data flow of an AI dialogue system based on multimodal input according to an embodiment of the present application.

[0011] Figure 3 It is a block diagram of an intention expression related feature extraction module in an AI dialogue system based on multimodal input according to an embodiment of the present application.

[0012] Figure 4 This is a block diagram of an intention expression image and text feature joint encoding module in an AI dialogue system based on multimodal input according to an embodiment of the present application. DETAILED DESCRIPTION

[0013] Below, the exemplary embodiments according to the present application will be described in detail with reference to the accompanying drawings. Obviously, the described embodiments are only part of the embodiments of the present application, rather than all the embodiments of the present application, and it should be understood that the present application is not limited to the exemplary embodiments described here.

[0014] The rapid development of artificial intelligence technology has greatly promoted the progress of natural language processing technology. This enables computers to understand and process human language more naturally, thus promoting the emergence and development of AI dialogue systems. Artificial intelligence dialogue systems use artificial intelligence technology to achieve human-computer communication. They can not only understand human language, but also imitate human conversation methods, providing users with a more intelligent and convenient interactive experience.

[0015] However, traditional AI dialogue systems usually rely solely on text input to understand and respond to user intent. This single modality understanding method may not fully capture the complexity and nuances of user expressions, which may cause the system to misunderstand the user's true intent and provide inaccurate responses, which can easily bring a bad experience to users.

[0016] In response to the above technical problems, the technical concept of this application is to first obtain the intention expression image and intention expression text input by the user, and then use artificial intelligence-based data analysis technology to extract image semantic features of the intention expression image and understand text semantics of the intention expression text, and then intelligently generate reply text based on the features and prompt words obtained by interactive fusion of the semantic coding features of the intention expression image and the semantic coding features of the intention expression text. In this way, by analyzing multimodal intention expression data, the AI ​​dialogue system can more accurately understand the user's actual intention, and then provide more accurate answers, which is conducive to improving user experience.

[0017] Figure 1 4 is a system block diagram of an AI dialogue system based on multimodal input according to an embodiment of the present application. Figure 2 Schematic diagram of data flow of an AI dialogue system based on multimodal input according to an embodiment of the present application. Figure 1 and Figure 2 As shown, in the AI ​​dialogue system 100 based on multimodal input, it includes: a user input data acquisition module 110, used to acquire first modal data and second modal data input by the user, wherein the first modal data is an intention expression image, and the second modal data is an intention expression text; an intention expression related feature extraction module 120, used to perform data analysis on the first modal data and the second modal data respectively to obtain semantic coding features of the intention expression image and semantic coding features of the intention expression text; an intention expression image text feature joint encoding module 130, used to perform multimodal principal component compensation interactive joint encoding on the semantic coding features of the intention expression image and the semantic coding features of the intention expression text to obtain intention expression image-text joint features; a reply text generation module 140, used to generate a reply text based on the intention expression image-text joint features and prompt words.

[0018] In an embodiment of the present application, the user input data acquisition module 110 is used to acquire the first modal data and the second modal data input by the user, wherein the first modal data is an intention expression image, and the second modal data is an intention expression text. It should be understood that the intention expression image can be any visual content that can convey the user's intention or background information, such as product pictures, scene photos, icons, etc. This image contains contextual information that cannot be fully described by words, such as the location, color, shape, expression or action of the person, etc. By analyzing the image, the system can obtain non-verbal clues about the user's intention, which helps to better understand the user's intention. The intention expression text specifically refers to the information provided by the user in the form of natural language, such as questions, commands, statements, etc. It directly expresses the user's query, demand or opinion, and usually includes elements such as keywords, phrases and sentence structures. The intention expression text data can provide the system with clear language information, help the system identify specific tasks, topics or emotional tendencies, and is important data for the system to parse user intentions. In general, the intention expression image provides intuitive visual information, which can help clarify the ambiguities in the text; while the intention expression text data supplements the specific details and background information that the image may lack. By combining and analyzing these two types of information, the model can understand user intent more accurately and generate more reasonable response text.

[0019] In the embodiment of the present application, the intention expression related feature extraction module 120 is used to perform data analysis on the first modal data and the second modal data to obtain the intention expression image semantic coding features and the intention expression text semantic coding features. Specifically, Figure 3 FIG. 1 is a block diagram of an intention expression related feature extraction module in an AI dialogue system based on multimodal input according to an embodiment of the present application. Figure 3 As shown, the intention expression related feature extraction module 120 includes: an intention expression image feature extraction unit 121, used to perform image semantic feature extraction on the first modal data to obtain the intention expression image semantic coding feature; an intention expression text semantic coding unit 122, used to perform text semantic understanding on the second modal data to obtain the intention expression text semantic coding feature.

[0020] In an embodiment of the present application, the intention expression image feature extraction unit 121 is used to extract image semantic features from the first modal data to obtain the intention expression image semantic coding features. Specifically, in an embodiment of the present application, the intention expression image feature extraction unit is used to: use an image processor based on the FCN model to extract image semantic features from the first modal data to obtain an intention expression image semantic coding feature vector as the intention expression image semantic coding feature. It should be understood that the intention expression image is not just a collection of pixels, but also contains rich semantic information, such as objects, scenes, actions, etc. These semantic information are crucial to understanding the user's intentions. For example, a user uploads a photo with a specific product, and the system needs to be able to identify the product and understand the user's possible questions or needs related to the product. In order to capture the semantic information in the original input image, in this application, it is necessary to use an image processor based on the FCN model to extract image semantic features from the first modal data to obtain an intention expression image semantic coding feature vector containing rich image semantic content. Those skilled in the art should understand that, unlike traditional convolutional neural networks, which use fully connected layers in the last few layers, resulting in a significant reduction in the spatial dimension of the output, FCN is composed entirely of convolutional layers and can preserve the spatial structure of the input image, which helps capture detailed information in the image, especially when it is necessary to focus on a specific area in the image. In addition, the FCN model can also integrate feature information at different levels through jump connections, which can capture global context information in the intended expression image while maintaining key local details, helping to improve the model's ability to understand the content of complex intended expression images.

[0021] The following is a detailed description of the specific implementation process of "using an image processor based on the FCN model to extract image semantic features from the first modality data to obtain the semantic encoding feature vector of the intent expression image": First, for the input intent expression image, data preprocessing operations are performed. Among them, the normalization operation of the image is crucial, and its purpose is to normalize the pixel values ​​of the image to a specific range, such as [0,1] or [-1,1]. The advantage of this is that when processing images from different sources, since their pixel value ranges may vary greatly, the normalization operation can eliminate these differences, making them comparable and consistent, and providing a unified data basis for the subsequent feature extraction process. At the same time, the image size needs to be adjusted. The FCN model usually has certain requirements for the size of the input image. For example, in many common implementations, the image is adjusted to a size such as 224×224 or 512×512. It should be understood that the architecture and parameters of the model are designed and trained based on a specific input size. If the size of the input image does not meet the requirements, the model may not work properly or the extracted features may not be effective. By adjusting the image to a suitable size, the image data structure can be better adapted to the convolutional layer, pooling layer, and upsampling layer of the FCN model, thereby ensuring the effectiveness of subsequent feature extraction.

[0022] Next is the architecture of the FCN model. In this architecture, there are multiple convolutional layers, pooling layers, and upsampling layers. In the convolutional layer, a series of convolutional kernels slide on the image to extract features at different levels through convolution operations. The initial convolutional layer focuses on extracting low-level features of the image, such as the edge and texture information of the image. These features are the basic building blocks of the image and are very important for subsequent higher-level semantic understanding. As the network goes deeper, subsequent convolutional layers will gradually extract more abstract semantic features, such as the local structure and shape of objects. The size and step size of the convolutional kernel are determined according to the specific FCN architecture. Common convolutional kernel sizes can be 3×3 or 5×5, with a step size of 1 or 2. The settings of these parameters will affect the refinement and range of the features extracted by the convolutional layer. Smaller convolutional kernels can extract finer details, while different step sizes will change the distance the convolutional kernel moves on the image, thereby affecting the density of feature extraction. The pooling layer generally uses maximum pooling or average pooling operations to reduce the size of the feature map while retaining the most significant information. The principle of maximum pooling is to select the maximum value in a small area as the representative of the area, so that while reducing the size of the feature map, the most significant local features can be highlighted. For example, in a 2×2 pooling area, the maximum pixel value is selected as the feature of this area, which helps to reduce the amount of data while retaining important feature information. Average pooling takes the average value and is suitable for different scenarios. The existence of the pooling layer gradually reduces the size of the feature map, while reducing the amount of calculation, allowing the network to focus on more important features. In the upsampling layer, its main function is to restore the feature map after multiple convolutions and pooling processes to a size similar to that of the input image. Transposed convolution (deconvolution) is usually used to implement this process, gradually restoring the low-resolution feature map to the size of the original image while maintaining semantic information. This step is very critical because it can re-enlarge the feature map that was reduced during the convolution and pooling process, and retain the semantic information extracted before, so that the final feature map not only contains abstract semantic information, but also corresponds to the spatial position of the original image, which can provide richer information for subsequent analysis and processing.

[0023] When the preprocessed intent expression image is input into the FCN model, it will go through the above-mentioned convolution, pooling and upsampling process. After being processed by multiple convolutional layers and pooling layers, the spatial dimension of the image will gradually decrease, while the feature dimension will continue to increase. In this process, various features from low-level to high-level will be extracted. These features are progressive, abstracted and refined layer by layer, and finally form a feature map containing rich semantic information. This feature map will contain various semantic information in the image, such as the category, position, and relationship of objects in the image. Finally, the final feature map needs to be flattened, which is the process of converting it from a two-dimensional or three-dimensional feature map to a one-dimensional vector. This one-dimensional vector is the final semantic encoding feature vector of the intent expression image, which encodes the semantic information in the image and contains various semantic elements expressed by the image, which can be used for subsequent fusion and analysis.

[0024] In actual implementation, deep learning frameworks such as PyTorch can be used to build such FCN models. For example, torchvision.transforms can be used to preprocess the input image, including resizing, converting to tensors, and normalization operations. Then, the FCN model is built based on an existing pre-trained model, such as ResNet50, and its last fully connected layer is replaced with a convolutional layer, and a suitable upsampling layer is added to implement the upsampling operation of the feature map. A random image can be input as an example, and it can be converted into a format that meets the model input requirements through preprocessing operations. The preprocessed image is input into the FCN model for feature extraction, and finally the extracted feature map is flattened through the view method to obtain the final semantic encoding feature vector of the intent expression image. It is worth noting that in actual applications, the above simple example is only a basic framework. In actual applications, the FCN model may need to be adjusted and optimized more deeply according to the specific situation. For example, the output channel can be adjusted according to the specific number of semantic categories, a more complex architecture or a different upsampling method can be used, etc. Moreover, in order for the FCN model to accurately extract semantic features in images, a large amount of image data is usually used for supervised training. By minimizing loss functions, such as cross entropy loss, the model parameters are optimized so that the model can better learn the semantic features of the image, thereby achieving better performance in image semantic feature extraction, so as to provide better quality image semantic information for the entire technical solution, thereby improving the entire system's ability to understand user intent and the quality of response.

[0025] In an embodiment of the present application, the intention expression text semantic encoding unit 122 is used to perform text semantic understanding on the second modal data to obtain the semantic encoding features of the intention expression text. Specifically, in an embodiment of the present application, the intention expression text semantic encoding unit is used to: after the second modal data is segmented, it is passed through a semantic understander based on a BERT model to obtain an intention expression text semantic encoding feature vector as the intention expression text semantic encoding feature. It should be understood that in order to segment the continuous intention expression text into meaningful units so that the model can more easily capture the semantic information of the original expression text, it is first necessary to perform word segmentation on the second modal data. It is worth mentioning that word segmentation can also reduce computational complexity and improve data processing efficiency. Then, the second modal data after word segmentation needs to be input into a semantic understander based on a BERT model for processing. It is known to those of ordinary skill in the art that BERT is a pre-trained language model that uses a large amount of unlabeled text data to learn the deep structure of the language. Unlike previous unidirectional language models, BERT uses a bidirectional encoding mechanism, which enables the model to capture the front-to-back dependencies of words in the intention expression text, which is conducive to capturing richer and more accurate semantic features. That is, through the semantic understander based on the BERT model, the system can obtain the semantic encoding feature vector of the intent expression text containing rich user intent context information. The information in this feature vector cannot be provided by the simple bag-of-words model or n-gram model. The generated semantic encoding feature vector of the intent expression text can more accurately capture the user's true intention, so that the system can give appropriate answers even when faced with ambiguous or complex intent expression text.

[0026] In an embodiment of the present application, the intention expression image text feature joint encoding module 130 is used to perform multimodal principal component compensation interactive joint encoding on the intention expression image semantic encoding features and the intention expression text semantic encoding features to obtain the intention expression image-text joint features. Specifically, in an embodiment of the present application, the intention expression image text feature joint encoding module is used to: perform intention expression image-text principal component analysis interactive fusion on the intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector to obtain the intention expression image-text joint encoding vector as the intention expression image-text joint feature. More specifically, Figure 4 FIG. 1 is a block diagram of an intention expression image text feature joint encoding module in an AI dialogue system based on multimodal input according to an embodiment of the present application. Figure 4As shown, the intention expression image text feature joint encoding module 130 includes: a feature principal component analysis unit 131, which is used to perform feature principal component analysis on the intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector to obtain a set of intention expression image semantic principal component encoding vectors and a set of intention expression text semantic principal component encoding vectors; a principal component aggregation encoding feature map construction unit 132, which is used to construct the set of intention expression image semantic principal component encoding vectors and the set of intention expression text semantic principal component encoding vectors into an intention expression image semantic principal component aggregation encoding feature map and an intention expression text semantic principal component aggregation encoding feature map. The invention relates to a sub-aggregated coding feature map; a feature map difference embedding compensation processing unit 133, which is used to perform difference embedding compensation processing on the semantic principal component aggregated coding feature map of the intended expression image and the semantic principal component aggregated coding feature map of the intended expression text to obtain the intended expression image-text semantic coding difference embedding compensation weight vector; an intended expression image-text joint coding feature generation unit 134, which is used to obtain the intended expression image-text joint coding vector based on the intended expression image-text semantic coding difference embedding compensation weight vector, the set of the intended expression image semantic principal component coding vectors and the set of the intended expression text semantic principal component coding vectors.

[0027] It should be understood that the semantic coding feature vector of the intention expression image is extracted from the intention expression image, and the semantic coding feature vector of the intention expression text is extracted from the intention expression text. These two feature vectors come from data from different sources and may have different distribution characteristics and scales. If the two feature vectors are simply spliced ​​together for fusion, the potential correlation between the two feature vectors may be ignored. In order to capture the potential correlation between the original intention expression image data and the text data, while ensuring that the image semantic coding features and the text semantic coding features can be balanced, so that the final fused feature vector retains the key part of the original data and can compensate for the difference information, in this application, it is necessary to perform the intention expression image-text principal component analysis interactive fusion on the intention expression image semantic coding feature vector and the intention expression text semantic coding feature vector to obtain the intention expression image-text joint coding vector as the intention expression image-text joint feature. By performing the intention expression image-text principal component analysis interactive fusion processing on these two vectors, the number of features can be effectively reduced, the computational complexity can be reduced, and the "dimensionality disaster" can be avoided.

[0028] Specifically, it is first necessary to perform feature principal component analysis on the semantic encoding feature vector of the intended expression image and the semantic encoding feature vector of the intended expression text to obtain a set of semantic principal component encoding vectors of the intended expression image and a set of semantic principal component encoding vectors of the intended expression text. The above process can be expressed as: ;in, and represent the semantic coding feature vector of the intention expression image and the semantic coding feature vector of the intention expression text respectively, Indicates passing The calculated covariance matrix of the semantic samples of the intent expression image is Indicates passing The calculated covariance matrix of the semantic samples of the intended expression text is: and They are the principal component orthogonal matrix of the intended expression image and the principal component orthogonal matrix of the intended expression text, and are the intended expression image diagonal matrix and the intended expression text diagonal matrix, and They are and The transposed matrix of for each intention expression image semantic principal component coding vector in the set of intention expression image semantic principal component coding vectors, for each intention expression text semantic principal component encoding vector in the set of the intention expression text semantic principal component encoding vectors, are the first and the second principal component encoding vectors of the intended expression image respectively. The weight value corresponding to the principal component encoding vector of the intended expression image semantics, are the first and the second principal component encoding vectors of the intended expression text, respectively. The weight value corresponding to the semantic principal component encoding vector of the intended expression text, and Respectively express and Perform principal component analysis of features.

[0029] It should be understood that the original intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector are high-dimensional, which increases the computational burden and may lead to overfitting problems. Dimensionality reduction through principal component analysis can significantly reduce the data dimension, thereby reducing the demand for computing resources, which is conducive to speeding up model training and reasoning. In addition, the principal component features obtained after principal component analysis are more interpretable, and each principal component feature represents an important change direction in the original data, which means that it is easier for the model to understand which feature information is important for expressing user intent, and then it can pay more attention to important information to improve the accuracy of understanding user intent.

[0030] Next, the set of the semantic principal component coding vectors of the intended expression image and the set of the semantic principal component coding vectors of the intended expression text are constructed into the intended expression image semantic principal component aggregation coding feature map and the intended expression text semantic principal component aggregation coding feature map. The above process can be expressed as: ;in, for each intention expression image semantic principal component coding vector in the set of intention expression image semantic principal component coding vectors, for each intention expression text semantic principal component encoding vector in the set of the intention expression text semantic principal component encoding vectors, represents the feature shape reshaping, and They respectively represent the semantic principal component aggregation coding feature map of the intention expression image and the semantic principal component aggregation coding feature map of the intention expression text.

[0031] It should be understood that when the set of principal component encoding vectors of the intended image semantics and the set of principal component encoding vectors of the intended text semantics are regarded as point clouds in a high-dimensional space, these point clouds do not exist in isolation, and there is an inherent correlation and coherence between them. By constructing them into principal component aggregation encoding feature maps of the intended image semantics and principal component aggregation encoding feature maps of the intended text semantics, this local neighborhood relationship can be maintained in a two-dimensional or three-dimensional space. For example, in the set of principal component encoding vectors of the intended image semantics, adjacent vectors may represent the features of adjacent regions in the image, or parts that are semantically similar. Mapping them into principal component aggregation encoding feature maps of the intended image semantics can ensure that these adjacent information in the high-dimensional space remain adjacent in the low-dimensional space, thereby maintaining the local structure and correlation between the information. At the same time, in the process of constructing the feature map, since the point cloud needs to be mapped to the low-dimensional space, the information will naturally be filtered and integrated. In this process, only those important and representative information in the local neighborhood will be retained, while some noise or less important information will be filtered or integrated, which determines which information will be retained for subsequent processing.

[0032] Specifically, in the embodiment of the present application, the feature map difference embedding compensation processing unit is used to: input the semantic principal component aggregation coding feature map of the intended expression image and the semantic principal component aggregation coding feature map of the intended expression text into the difference embedding unit to obtain the semantic coding weight vector of the intended expression image and the semantic coding weight vector of the intended expression text. The process can be expressed as: ;in, and They respectively represent the semantic principal component aggregation coding feature map of the intention expression image and the semantic principal component aggregation coding feature map of the intention expression text, Indicates the average pooling operation. and denote the first intention expression image weight matrix and the second intention expression image weight matrix respectively, represents a nonlinear activation function, represents the output activation function, represents the semantic encoding weight vector of the intended expression image, and denote the first intention expression text weight matrix and the second intention expression text weight matrix respectively, Represents the semantic encoding weight vector of the intended expression text.

[0033] The intention expression image-text semantic coding difference embedding compensation weight vector is calculated between the intention expression image semantic coding weight vector and the intention expression text semantic coding weight vector. The process can be expressed as: ;in, represents the semantic encoding weight vector of the intended expression image, represents the semantic encoding weight vector of the intended expression text, It means to subtract by position point. Indicates the calculation of absolute value. Represents the intention expression image-text semantic encoding difference embedding compensation weight vector.

[0034] It should be understood that in order to capture the difference between the semantic principal component aggregation coding feature map of the intended expression image and the semantic principal component aggregation coding feature map of the intended expression text, and at the same time quantify the importance of these differences, it is necessary to input the semantic principal component aggregation coding feature map of the intended expression image and the semantic principal component aggregation coding feature map of the intended expression text into the difference embedding unit for processing to generate the semantic encoding weight vector of the intended expression image and the semantic encoding weight vector of the intended expression text that reflect the relative importance of the features at each position. The two generated weight vectors are similar to adjustment factors. They ensure that fair interaction and fusion can be achieved even when there are significant differences in the original features. That is, no matter which modality has richer or more representative data, it can be properly valued, thereby avoiding any modality from having too much influence on the final result. Then in order to make up for the possible information asymmetry, that is, when the information provided by one modality is not enough to fully describe a certain semantic concept, the other modality can supplement the missing part through the role of the compensation weight vector, so that the overall understanding is more comprehensive and accurate. In this application, it is necessary to calculate the intended expression image-text semantic encoding difference embedding compensation weight vector between the semantic encoding weight vector of the intended expression image and the semantic encoding weight vector of the intended expression text. In this way, the system can improve its understanding of complex scenarios and provide a deeper semantic analysis of user intent.

[0035] Specifically, in the embodiment of the present application, the intention expression image-text joint coding feature generation unit is used to: calculate the position mean vector of the set of the intention expression image semantic principal component coding vectors and the set of the intention expression text semantic principal component coding vectors to obtain the intention expression image semantic feature principal component representation coding vector and the intention expression text semantic feature principal component representation coding vector. The process can be expressed as: ;in, and The first and second principal component encoding vectors of the set of principal component encoding vectors of the intended image and the set of principal component encoding vectors of the intended text are represented respectively. The principal component encoding vector of the intention expression image semantics and the The semantic principal component encoding vector of the intent expression text, The number of vectors in the set of principal component encoding vectors that represent the semantics of the image and the set of principal component encoding vectors that represent the semantics of the text, and They respectively represent the principal component representation encoding vector of the semantic features of the intention-expressing image and the principal component representation encoding vector of the semantic features of the intention-expressing text.

[0036] Based on the intention expression image-text semantic coding difference embedding compensation weight vector, the intention expression image semantic feature principal component representation coding vector and the intention expression text semantic feature principal component representation coding vector are aggregated and interacted to obtain the intention expression image-text joint coding vector. This process can be expressed as: ;in, and They respectively represent the principal component representation encoding vector of the semantic features of the intended expression image and the principal component representation encoding vector of the semantic features of the intended expression text, It means point multiplication by position. represents the intention expression image-text semantic encoding difference embedding compensation weight vector, represents a nonlinear activation function, represents point convolution, Represents the intention expression image-text joint encoding vector.

[0037] It should be understood that by calculating the position mean vector, the amount of data to be processed can be greatly reduced before the aggregation interaction. This not only improves the processing efficiency, but also reduces the demand for computing resources. Different from the traditional principal component analysis method, it focuses on the average of the eigenvalues ​​at the same position. It uses this method to achieve data compression and is suitable for multimodal data fusion scenarios, aiming to reduce redundancy while maintaining core information. Finally, it is necessary to embed the compensation weight vector based on the intention expression image-text semantic coding difference to perform aggregation interaction processing on the principal component representation coding vector of the semantic feature of the intention expression image and the principal component representation coding vector of the semantic feature of the intention expression text. Here, the intention expression image-text semantic coding difference embedding compensation weight vector guides the way to adjust the interaction according to the difference between the semantic coding feature vector of the intention expression image and the semantic coding feature vector of the intention expression text. The aggregation interaction not only integrates the information from the two original intention expression images and the intention expression text features, but also ensures the sensitivity and adaptability of the intention expression image-text joint coding vector generated by the interaction to the original feature differences by introducing a compensation mechanism. This means that even if there are subtle differences in the semantic and visual information between images and texts, the system can handle them properly, discover clues to user intent hidden in the differences, and improve its ability to parse complex user intent.

[0038] In an embodiment of the present application, the reply text generation module 140 is used to generate a reply text based on the intention expression image-text joint features and prompt words. Specifically, in an embodiment of the present application, the reply text generation module is used to: after adding the prompt word at the end of the intention expression image-text joint encoding vector, input it into the dialogue module based on the large language model to obtain the reply text. It should be understood that the prompt word is a pre-designed text that provides a context or instruction for the dialogue system to better guide the model to generate an answer that meets the expectations. The prompt word may contain specific instructions, question types, tone style and other information. Specifically, although the intention expression image-text joint encoding vector contains feature information extracted from the intention expression image and the intention expression text, they may not be enough to clearly indicate how the model generates a response. The prompt word can provide additional guidance to tell the model what type of answer the user wants, or how the reply should be constructed. Here, the large language model specifically refers to those artificial intelligence models that have been trained with a large corpus and have a large number of parameters. It often has powerful natural language processing capabilities and can generate coherent and meaningful text output based on a given context. By inputting the intent expression image-text joint encoding vector with the prompt word into the large language model, the context understanding and generation capabilities of the large language model can be fully utilized to generate the expected reply text. In this way, not only can the relevance and accuracy of the reply be improved, but also the generated text can be ensured to be natural and fluent, easy for users to understand and accept.

[0039] The following is a detailed description of the specific implementation process of "after adding the prompt word at the end of the intention expression image-text joint encoding vector, input it into the dialogue module based on the large language model to obtain the reply text": First, in the data preparation stage, it is necessary to obtain the intention expression image-text joint encoding vector and determine the prompt word. For the determination of the prompt word, this requires in-depth consideration of the application scenario and the specific needs of the user. For example, in an educational tutoring scenario, if a student uploads a picture of a math problem and attaches a text description of the puzzle point in solving the problem, the prompt word may be set as "for the math problem picture and text description provided by the student, explain the solution ideas and methods in detail"; in a food recommendation scenario, if the user shows a photo of a dinner party and mentions the taste preference, the prompt word can be "based on the user's dinner party picture and taste preference, recommend suitable food dishes and restaurants". After determining the prompt word, in order to effectively combine the prompt word with the joint encoding vector, it is necessary to use the word embedding mechanism of the large language model itself or the pre-trained word embedding model (such as GloVe, Word2Vec, etc.) to convert the prompt word into a vector form. During this conversion process, it is necessary to ensure that the dimension of the converted vector matches the joint encoding vector or can be smoothly spliced. This may involve operations such as vector dimension adjustment and format conversion to ensure data consistency and compatibility.

[0040] Next is the preparation for inputting the large language model. First, you need to select a suitable large language model, which requires comprehensive consideration of the application requirements, available computing resources, and performance expectations. For example, models such as GPT-3 and GPT-4 have excellent performance, but they also have high requirements for computing resources; while T5 or some open source lightweight large language models provide a viable option when resources are limited. After selecting the model, load the pre-trained model and the corresponding word segmenter from the corresponding repository (such as the Hugging FaceTransformers library). When constructing the input, add the encoded prompt word to the tail of the joint encoding vector. At the same time, in order to help the large language model better understand the structure and semantics of the input, it is also necessary to add start tags (such as <|InnerThoughtBegin|>) and end tags (such as <|InnerThoughtEnd|>) according to the requirements of the model. These tags are a kind of guiding signal to the model, so that it can accurately identify the boundaries and intentions of the input.

[0041] Finally, the large language model is called to generate replies. Before this, the concatenated vectors need to be converted into the input format required by the large language model. After the input format adjustment is completed, the data is input into the inference function or API of the large language model. During the call process, some key parameters need to be set to control the generation process. For example, the maximum length of the generated text determines the length of the reply. Reasonable settings can avoid generating replies that are too long or too short; the diversity of generation can be achieved by adjusting the temperature parameter. A lower temperature will make the generated replies more certain and concentrated, which is suitable for scenarios that require precise answers, while a higher temperature will increase the diversity and randomness of the replies, which is suitable for situations that require creativity and multiple possibilities; the number of generated samples can provide multiple replies for subsequent screening and evaluation. After receiving the input, the large language model will perform forward propagation based on its internal complex neural network structure and a large amount of pre-trained knowledge, deeply analyze and understand the input semantic information, and predict the subsequent text content based on its own language generation mechanism. The replies generated by the model are usually presented in the form of original text sequences, which may contain some special tags or redundant information. Therefore, these replies need to be further processed to remove special marks, filter out the most relevant and valuable replies, and post-process the text, such as cleaning up extra spaces, line breaks, etc., so as to obtain the final reply text that is clear, accurate and meets user needs.

[0042] As a practical example, suppose that an intelligent home decoration design assistant is being developed. The user uploads a photo of the living room and describes the decoration style and functional requirements that he or she likes in the text. First, the intent expression image-text joint encoding vector is obtained through the previous image and text processing process, and then the prompt word is determined to be "Generate a detailed home decoration design plan based on the living room photo and decoration requirements provided by the user". After converting the prompt word into a vector and concatenating it with the joint encoding vector, a suitable large language model (such as the GPT-3 model that has been fine-tuned in the home decoration field) is selected, and the format is adjusted and input according to the input requirements of the model. After receiving the input, the model generates a series of reply texts through complex calculations and reasoning. These reply texts may include different design plans, furniture recommendations, and layout suggestions. These replies are screened and processed to remove irrelevant information and repeated content, and finally a clear and practical home decoration design plan can be provided to the user as a reply text.

[0043] Throughout the implementation process, many practical factors need to be considered. From the perspective of performance and resources, the operation of large language models usually requires a lot of computing resources, especially when processing large-scale user requests. Therefore, it may be necessary to use GPU acceleration technology or a distributed computing architecture to improve processing speed and efficiency to ensure that the system can quickly respond to user requests. For online service applications, effective load balancing and resource management are also required to reasonably allocate computing resources to avoid system crashes or response delays due to high concurrent requests. In the application of the model, in order to better adapt to specific application areas and business needs, it is often necessary to fine-tune the large language model. By collecting and organizing data in a specific field, the model is trained in a targeted manner so that it can better understand and process the professional terms and semantic relationships in the field, thereby generating responses that are more in line with actual needs. In addition, security and ethical considerations cannot be ignored. In terms of filtering input data and reviewing output responses, strict mechanisms need to be established to prevent the generation of harmful, discriminatory or misleading content and ensure that the use of the system complies with legal and ethical standards. Finally, user experience is always the core goal. We need to continuously optimize the design of prompt words, adjust the parameter settings of the model, and improve the entire processing flow based on user feedback to continuously improve the quality and accuracy of responses, meet user expectations, and enhance user satisfaction and trust in the system. In short, by continuously optimizing each link of this process, we can give full play to the advantages of multimodal information and large language models to provide users with efficient and intelligent services and assistance.

[0044] In particular, since the intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector respectively represent the image semantic features of the first modal data and the text semantic features of the second modal data, when the feature principal component compensation interaction is performed between them, the semantic feature population attributes of different modal data will have fairness differences in the principal component compensation level, thereby affecting the feature distribution interaction inclusiveness of the intention expression image-text joint encoding vector, and reducing the text quality of the reply text obtained by inputting the dialogue module based on the large language model.

[0045] Preferably, after adding a prompt word to the end of the intention expression image-text joint encoding vector, when it is input into the dialogue module based on the large language model to obtain the reply text, the intention expression image-text joint encoding vector is optimized, specifically including: determining the mean of the eigenvalues ​​corresponding to the intention expression image-text joint encoding vector , and the intention expression image-text joint encoding vector After adding the mean of the eigenvalues, the result of the addition is multiplied by the mean of the eigenvalues ​​to obtain the conditional probability constraint vector of the intended expression image-text joint encoding. ,in, It means adding by position point, that is, adding the corresponding value to the eigenvalue of each position of the vector. Represents point product by position.

[0046] Determine the standard deviation of the eigenvalues ​​corresponding to the intention expression image-text joint encoding vector , and further perform point-wise subtraction of the intention expression image-text joint coding conditional probability constraint vector and the eigenvalue standard deviation vector from the intention expression image-text joint coding vector to obtain the intention expression image-text joint coding interactive fairness measurement vector ,in, Indicates point subtraction by position.

[0047] The bit-by-bit inverse of the intention expression image-text joint coding interactive fairness measurement vector is added to the intention expression image-text joint coding conditional probability constraint vector, and then multiplied with the intention expression image-text joint coding vector to obtain the intention expression image-text joint coding group feature transfer vector ,in, The bit-by-bit reciprocal of the intention-expression image-text joint coding interactive fairness metric vector is represented, that is, the reciprocal of the eigenvalue of each position of the intention-expression image-text joint coding interactive fairness metric vector is calculated.

[0048] The eigenvalue mean The standard deviation of the eigenvalue The sum is multiplied by the scaling weight hyperparameter Then, it is multiplied by the intention expression image-text joint coding group feature transfer vector to obtain the optimized intention expression image-text joint coding vector .

[0049] After adding the prompt word at the end of the optimized intention expression image-text joint encoding vector, it is input into the dialogue module based on the large language model to obtain the reply text.

[0050] That is, in order to solve the problem of attribute balance deviation of the fusion features of the intention expression image-text joint coding vector in cross-modal samples, it is necessary to enhance the dynamic adaptability of the representation space heterogeneity of the intention expression image-text joint coding vector. Therefore, by introducing an interactive fairness measurement model constructed based on conditional probability constraints for the intention expression image-text joint coding vector, the group feature transfer mechanism of the intention expression image-text joint coding vector is iteratively optimized, and the statistical association mechanism is adopted as the adjustment factor of hierarchical fairness compensation.

[0051] In this way, a robust distribution consistency representation is achieved through the multidimensional feature collaborative framework of the intention expression image-text joint coding vector, and finally a feature interaction paradigm with adaptive adjustment capability is formed, which effectively improves the fairness and collaborative efficiency of the multimodal joint coding of the intention expression image-text joint coding vector, and improves the text quality of the reply text obtained by the dialogue module based on the large language model as input.

[0052] In summary, the AI ​​dialogue system 100 based on multimodal input according to the embodiment of the present application is explained, which uses artificial intelligence-based data analysis technology to extract image semantic features of the intention expression image input by the user and to understand the text semantics of the intention expression text input by the user, and intelligently generates a reply text based on the features and prompt words obtained by interactively fusion of the semantic coding features of the intention expression image and the semantic coding features of the intention expression text by principal component analysis. In this way, by analyzing the multimodal intention expression data, the AI ​​dialogue system can more accurately understand the actual intention of the user, and then provide a more accurate answer, which is conducive to improving the user experience.

Claims

1. An AI dialogue system based on multimodal input, characterized in that: include: A user input data acquisition module is used to acquire first modal data and second modal data input by the user, wherein the first modal data is an intention expression image and the second modal data is an intention expression text; an intention expression related feature extraction module is used to perform data analysis on the first modal data and the second modal data respectively to obtain semantic coding features of the intention expression image and semantic coding features of the intention expression text; an intention expression image text feature joint encoding module is used to perform multimodal principal component compensation interactive joint encoding on the semantic coding features of the intention expression image and the semantic coding features of the intention expression text to obtain intention expression image-text joint features; a reply text generation module is used to generate a reply text based on the intention expression image-text joint features and prompt words.

2. The AI ​​dialogue system based on multimodal input according to claim 1, characterized in that: The intention expression related feature extraction module includes: an intention expression image feature extraction unit, which is used to extract image semantic features of the first modal data to obtain the intention expression image semantic coding features; and an intention expression text semantic coding unit, which is used to perform text semantic understanding on the second modal data to obtain the intention expression text semantic coding features.

3. The AI ​​dialogue system based on multimodal input according to claim 2, characterized in that: The intention expression image feature extraction unit is used to: use an image processor based on the FCN model to extract image semantic features from the first modality data to obtain an intention expression image semantic coding feature vector as the intention expression image semantic coding feature.

4. The AI ​​dialogue system based on multimodal input according to claim 3, characterized in that: The intention expression text semantic encoding unit is used to: segment the second modal data and then pass it through a semantic understander based on the BERT model to obtain an intention expression text semantic encoding feature vector as the intention expression text semantic encoding feature.

5. The AI ​​dialogue system based on multimodal input according to claim 4, characterized in that: The intention expression image-text feature joint encoding module is used to: perform intention expression image-text principal component analysis interactive fusion on the intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector to obtain the intention expression image-text joint encoding vector as the intention expression image-text joint feature.

6. The AI ​​dialogue system based on multimodal input according to claim 5, characterized in that: The intention expression image and text feature joint encoding module includes: a feature principal component analysis unit, which is used to perform feature principal component analysis on the intention expression image semantic encoding feature vector and the intention expression text semantic encoding feature vector to obtain a set of intention expression image semantic principal component encoding vectors and a set of intention expression text semantic principal component encoding vectors; a principal component aggregation encoding feature map construction unit, which is used to construct the set of intention expression image semantic principal component encoding vectors and the set of intention expression text semantic principal component encoding vectors into an intention expression image semantic principal component aggregation encoding feature map and an intention expression text semantic principal component aggregation encoding feature map; a feature map difference embedding compensation processing unit, which is used to perform difference embedding compensation processing on the intention expression image semantic principal component aggregation encoding feature map and the intention expression text semantic principal component aggregation encoding feature map to obtain an intention expression image-text semantic encoding difference embedding compensation weight vector; an intention expression image-text joint encoding feature generation unit, which is used to obtain the intention expression image-text joint encoding vector based on the intention expression image-text semantic encoding difference embedding compensation weight vector, the set of the intention expression image semantic principal component encoding vectors and the set of the intention expression text semantic principal component encoding vectors.

7. The AI ​​dialogue system based on multimodal input according to claim 6, characterized in that: The feature map difference embedding compensation processing unit is used to: input the intention expression image semantic principal component aggregation coding feature map and the intention expression text semantic principal component aggregation coding feature map into a difference embedding unit to obtain an intention expression image semantic coding weight vector and an intention expression text semantic coding weight vector; calculate the intention expression image-text semantic coding difference embedding compensation weight vector between the intention expression image semantic coding weight vector and the intention expression text semantic coding weight vector.

8. The AI ​​dialogue system based on multimodal input according to claim 7, characterized in that: The intention expression image-text joint coding feature generation unit is used to: calculate the positional mean vector of the set of the intention expression image semantic principal component coding vectors and the set of the intention expression text semantic principal component coding vectors to obtain the intention expression image semantic feature principal component representation coding vector and the intention expression text semantic feature principal component representation coding vector; based on the intention expression image-text semantic coding difference embedding compensation weight vector, aggregate and interact the intention expression image semantic feature principal component representation coding vector and the intention expression text semantic feature principal component representation coding vector to obtain the intention expression image-text joint coding vector.

9. The AI ​​dialogue system based on multimodal input according to claim 8, characterized in that: The reply text generation module is used to: after adding the prompt word at the end of the intention expression image-text joint encoding vector, input it into the dialogue module based on the large language model to obtain the reply text.

Images