Text processing feedback system and method based on multi-modal data

Abstract

The invention relates to the technical field of multi-modal data, and discloses a text processing feedback system and method based on multi-modal data, and the method comprises the steps: processing text data input by a user through a context encoder containing an embedded layer, and obtaining semantic feature vectors of a plurality of texts; arranging the plurality of semantic feature vectors into a one-dimensional feature vector; the method comprises the following steps: processing image data provided by a user through a convolutional neural network, and extracting a key feature vector in an image; calculating the product between the transpose of the image understanding feature vector and the text understanding feature vector to obtain a multi-modal demand expression matrix; adjusting each characteristic value in the matrix through a characteristic correction mechanism based on the characteristic value of each position in the multi-modal demand expression matrix; and inputting the corrected multi-modal demand expression matrix into a multi-label classifier for processing. The method has the advantage of improving the comprehensive text generation capability.

Description

Technical Field

[0001] The present invention relates to the technical field of multimodal data, and in particular to a text processing feedback system and method based on multimodal data. Background Art

[0002] With the rapid development of artificial intelligence and natural language processing technology, text generation and content creation are increasingly widely used in various fields, covering multiple scenarios such as automatic writing, intelligent customer service, advertising creativity, content recommendation, etc. Traditional text generation methods mainly rely on rule engines or models based on a single modality, such as pure text input and output. These methods usually rely on a large number of manually written rules or statistical models based on specific corpora to generate text. Although they can generate content that conforms to grammatical norms, they are relatively limited in semantic richness and generation diversity. Multimodal technology can fuse information from different sensors or data sources to generate more accurate and rich content. For example, combined with the emotional information of voice input and the visual information of image background, it can not only understand the user's language, but also analyze the input content from multiple dimensions such as context and emotion to generate more personalized text. This multimodal data fusion greatly improves the quality and intelligence level of text generation, especially in complex application scenarios such as customer service dialogues and creative writing.

[0003] However, existing technologies often lack comprehensive analysis and intelligent fusion of different data sources. When faced with multimodal input, text generation systems find it difficult to efficiently combine information from different modalities, resulting in certain limitations on the accuracy and diversity of generated content. Especially in some scenarios that require high-precision content creation, traditional methods may not be able to meet the personalized needs of users, and the generated text content is relatively simple in terms of style, tone, emotion, etc., lacking flexibility and diversity. Therefore, it is necessary to design a text processing feedback system and method for multimodal data that improves the comprehensive ability of text generation. Summary of the invention

[0004] In view of the deficiencies in the prior art, the present invention provides a text processing feedback system and method based on multimodal data, which has the advantages of assisting users in generating high-quality content, supporting writing in a variety of genres and styles, meeting the needs of different scenarios, and solving the problems in the above-mentioned background technology.

[0005] In order to achieve the above-mentioned purpose of assisting users to generate high-quality content, supporting writing in a variety of genres and styles, and meeting the needs of different scenarios, the present invention provides the following technical solutions: a text processing feedback system based on multimodal data, including a multimodal data input module, a text semantic feature extraction module, a multi-scale feature extraction module, an image understanding module, a multimodal demand expression matrix generation module, a feature correction module and a text generation optimization module; The multimodal data input module is used to receive multimodal data input by a user, including text, image, voice or video; The text semantic feature extraction module is used to extract semantic feature vectors from the text input by the user; The multi-scale feature extraction module is used to perform multi-scale feature extraction on text semantic features; The image understanding module is used to extract image feature vectors from image data provided by a user; The multimodal demand expression matrix generation module is used to fuse text features and image features to generate a multimodal demand expression matrix; The feature correction module is used to correct the features in the multimodal demand expression matrix; The text generation optimization module generates text content based on the corrected multimodal demand expression matrix and optimizes it according to user feedback.

[0006] The text processing feedback method based on multimodal data includes the following steps: S1: The text data input by the user is processed through a context encoder containing an embedding layer to obtain semantic feature vectors of multiple texts. Natural language processing technology is used to deeply understand the text and capture the semantic information in it. Preferably, S1 further includes using a pre-trained word embedding model to convert words or subwords into vector representations of fixed dimensions, and dynamically generating word vectors through the model to capture contextual information. BERT uses bidirectional encoding to capture information in the text from different angles through multiple attention heads. The model selectively focuses on different parts of the input sequence through a self-attention mechanism, integrates information at all positions, generates deep semantic features, performs weighted averaging or pooling on word vectors in sentences or documents, generates text-level semantic representations, and strengthens word embeddings through a context encoder to generate word vectors containing contextual information. Multiple word vectors are aggregated to obtain semantic vectors for sentences or paragraphs, and features of multiple layers are extracted through a multi-level encoder.

[0007] S2: Arrange multiple semantic feature vectors into a one-dimensional feature vector and process the vector through a multi-scale neighborhood feature extraction module; Preferably, S2 further includes integrating semantic feature vectors extracted from multiple texts into a unified representation, sequentially splicing multiple semantic feature vectors into a long vector, and performing multi-scale processing on the spliced ​​feature vectors to capture semantic information at different levels.

[0008] S3: Process the image data provided by the user through a convolutional neural network to extract the key feature vectors in the image; Preferably, S3 further includes using a convolution layer to extract low-level features in the image, sliding convolution kernels of different sizes on the image to extract features of local areas, each feature map corresponds to a specific pattern in the image, and the convolutional neural network includes multiple convolution layers, each layer using convolution kernels of different sizes.

[0009] S4: Calculate the product of the transpose of the image understanding feature vector and the text understanding feature vector to obtain a multimodal demand expression matrix; Preferably, the S4 further comprises: The image understanding feature vector is , whose dimension is 1*m, and the calculation formula is: ; In the formula, is the i-th value of the image feature vector; The text understanding feature vector is , whose dimension is 1*n, and the calculation formula is: ; In the formula, is the jth value of the text feature vector; Understanding feature vectors for images Perform a transposition operation to transform it from a 1*m row vector to an m*1 column vector, and transpose the image feature vector With text feature vector Multiply them together to get an m*n matrix. The calculation formula is: ; Matrix multiplication is calculated as follows: is an m*1 column vector, is a 1*n row vector; Results Matrix The dimension is m*n, and each element represents the correlation between the image and the text in the corresponding dimension; The element calculation formula is: ; Where i=1,...,m and j=1,...,n, indicating that the calculation of each element in the matrix is ​​the element-by-element product of image features and text features.

[0010] S5: Based on the eigenvalues ​​of each position in the multimodal demand expression matrix, each eigenvalue in the matrix is ​​adjusted through a feature correction mechanism; Preferably, S5 further includes adjusting the eigenvalues ​​in the matrix through a feature correction mechanism, and calculating the importance of the text features and the image features, and the calculation formula is: ; In the formula, is the weight of the i-th feature of the text, reflecting the importance of the text feature. is the i-th text feature. TF-IDF is a weight evaluation method based on text frequency and document frequency, which measures the relative importance of each word. The edge strength in the image is used as the weight of the image feature, and the calculation formula is: ; In the formula, is the weight of the jth feature of the image, reflecting the importance of the image feature. is the intensity corresponding to image feature j, is the maximum intensity of all features in the image; Each element in the multimodal demand expression matrix is ​​corrected by the text feature weight and the image feature weight. Each eigenvalue in the matrix is ​​weighted by the corresponding text and image feature weights. The calculation formula of the corrected matrix element is: ; In the formula, is the element in the original matrix, representing the relationship between text feature i and image feature j, is the corrected eigenvalue, after weighted adjustment, It reflects the combined contribution of text and image features.

[0011] S6: input the corrected multimodal demand expression matrix into a multi-label classifier for processing, and generate a set of classification results through the reasoning of the classifier, which represents the topic labels of the user input request; Preferably, the S6 further includes that the corrected multimodal demand expression matrix will be passed to the classifier for further feature mapping, and the classifier will fuse and map the input features. For the deep learning model, the feature mapping is implemented by a multi-layer neural network, and the neural network learns the complex relationship between features through multiple hidden layers; the core function of the classifier is to process the input multimodal features and generate a predicted probability for each label. For each label, the classifier will predict a probability value to indicate the relevance of the label to the user request. The sigmoid activation function is used in the last layer to output an independent probability value for each label, indicating whether the label is related to the input request. The calculation formula is: ; In the formula, (·) is the sigmoid function, is the input matrix The i-th type output.

[0012] S7: Generate high-quality text content that meets user needs based on the classification results and the identified topic tags.

[0013] Preferably, S7 further includes generating text content related to the topic tag through a generation model according to the output of the classifier. During the generation process, the input classification label is used as a condition to guide the text generation model to output text content that meets user needs.

[0014] Compared with the prior art, the present invention provides a text processing feedback system and method based on multimodal data, which has the following beneficial effects: The present invention obtains a multimodal demand expression matrix by calculating the product of image features and text features, realizes the fusion of semantic information of text and image in a unified space, can effectively integrate multimodal information, and create a comprehensive matrix that expresses user needs, thereby laying the foundation for subsequent reasoning and classification. By correcting each eigenvalue of the multimodal demand expression matrix and adjusting the weight of the fusion feature, data of different modes can work better together. By generating the subject label of the user's request through classification reasoning, the subject represented by the combination of text and image can be accurately identified, providing a basis for generating related content. The multi-label classifier can also identify multiple related subject labels, enhance the flexibility and adaptability of the system, and make the generated content more in line with the diverse needs of users. BRIEF DESCRIPTION OF THE DRAWINGS

[0015] Figure 1 It is a schematic diagram of the structure of the present invention; Figure 2 Schematic diagram of the method of the present invention. DETAILED DESCRIPTION

[0016] The following will be combined with the drawings in the embodiments of the present invention to clearly and completely describe the technical solutions in the embodiments of the present invention. Obviously, the described embodiments are only part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0017] The present invention provides a technical solution: a text processing feedback method based on multimodal data, such as Figure 1-2 As shown, the following steps are included: S1: The text data input by the user is processed through a context encoder containing an embedding layer to obtain semantic feature vectors of multiple texts. Natural language processing technology is used to deeply understand the text and capture the semantic information in it. Convert words or subwords in the text into fixed-dimensional vector representations. Use pre-trained word embeddings, such as Word2Vec, GloVe, or FastText, to map each word to a high-dimensional vector space. Use more advanced models that dynamically generate word vectors based on the role of words in context, capturing more contextual information. Encode the text, capture contextual information, and generate more accurate semantic representations. The encoder based on the Transformer model uses the self-attention mechanism to capture long-distance dependencies and contextual information. BERT uses bidirectional encoding, that is, learning the semantics of the text from left to right and from right to left at the same time, so that the contextual information of the text is fully utilized. Through multiple attention heads, the Transformer can capture information in the text from different angles and improve the understanding of context and semantics. Through the self-attention mechanism, the model can selectively focus on different parts of the input sequence to better understand which words or subwords are critical to the current semantics. At each position of the input, the model integrates the information of all other positions into the representation of the current word, thereby obtaining the deep semantic features of the context. By weighted averaging or pooling the feature vectors of all words in a sentence or document, a text-level semantic representation is generated for subsequent classification, generation or other tasks. Generate a high-dimensional vector representation that reflects the meaning of the text as the input of the subsequent processing module. The embedding of each word can be strengthened by the context encoder to generate a word vector containing context information. Multiple word vectors of the text are aggregated to obtain the semantic vector of the sentence or paragraph. The features of multiple layers are extracted through a multi-level encoder to obtain richer semantic information, which is suitable for the needs of different tasks.

[0018] S2: Arrange multiple semantic feature vectors into a one-dimensional feature vector and process the vector through a multi-scale neighborhood feature extraction module; Integrate the semantic feature vectors extracted from multiple texts into a unified representation. Concatenate multiple semantic feature vectors in order into a long vector. This process ensures that the relationship between different semantic information is not lost. Perform a pooling operation on the feature vectors. For example, perform maximum pooling or average pooling on multiple word vectors to obtain a fixed-length vector representation that represents the core semantics of the entire sentence or paragraph. Perform multi-scale processing on the concatenated feature vectors to capture semantic information at different levels and extract richer features. Use different processing scales or fields of view to extract information from different levels and understand the semantics of the text from both local and global dimensions, including: Local feature extraction: Use smaller windows to extract local features of sentences or paragraphs, which focus on short-term dependencies.

[0019] Global feature extraction: Use a larger window to extract global features in the text, which focus on long-term dependencies and the overall meaning of the sentence; Through different scales of vision, the concatenated text feature vector is processed to extract more comprehensive semantic information. Multiple convolutional layers are used, and the convolution kernel size of each convolutional layer is different to capture information of different scales. For example, a convolutional layer with a convolution kernel size of 2 is used to extract 2-gram features, and a convolutional layer with a convolution kernel size of 3 is used to extract 3-gram features. The feature dimension is further compressed through the pooling layer while maintaining important feature information. Pooling operations of different scales are used, such as local pooling for small windows and global pooling for large windows. Multi-head self-attention mechanism is used to enhance the fusion of multi-scale features. Different attention heads capture different levels of dependencies of text. For example, short-distance dependencies are captured by one attention head, while long-distance dependencies are captured by another attention head. The feature information extracted at different scales is fused into a unified feature representation. The features extracted at different scales are fused into a final feature representation through splicing, weighted averaging or other appropriate operations. The feature vectors extracted by different convolution kernels are spliced ​​or weighted summed to obtain a unified, multi-scale fused feature vector. The fused feature vector is converted into the final high-dimensional feature representation through a fully connected layer or other further processing, ready for use in downstream tasks.

[0020] S3: Process the image data provided by the user through a convolutional neural network to extract the key feature vectors in the image; Use convolutional layers to extract low-level features in images. Slide convolution kernels of different sizes on the image to extract features of local areas. The result of the convolution operation is a feature map. Each feature map corresponds to a specific pattern in the image. The convolutional neural network includes multiple convolutional layers. Each layer uses convolution kernels of different sizes to extract features at different levels. The first layer captures simple edges or textures, the second layer captures more complex shapes or patterns, and subsequent layers capture higher-level abstract features. After each convolutional layer, the ReLU activation function is applied to convert negative values ​​to zero and keep positive values ​​unchanged, which helps to improve the nonlinear expression ability of the model and avoid the gradient vanishing problem; Select the maximum value from each pooling window and retain the most significant features. This is usually used to reduce the spatial size and the amount of calculation. The average value is calculated from the pooling window as the feature of the area. The pooling layer is usually placed after the convolution layer, which can effectively reduce the number of parameters and the amount of calculation, while preventing overfitting and enhancing the model's robustness to displacement and deformation; Through multi-layer convolution and pooling operations, high-level features of the image are gradually extracted. As the number of convolution layers increases, the network gradually shifts from simple edge detection to complex object feature recognition. For example, the first layer may capture simple edges, the second layer may capture corners and textures, and the subsequent layers may capture object parts or overall shapes. Multi-layer convolution and pooling can help the network gradually extract high-level complex features from low-level features, and ultimately identify key objects and scene information in the image; The high-level features extracted by the convolution and pooling layers are converted into one-dimensional feature vectors for further classification or regression tasks. The multi-dimensional feature map is flattened into a one-dimensional vector to prepare data for the fully connected layer input. One or more fully connected layers are applied to the flattened data to map the local features to a high-dimensional semantic space to further extract high-level semantic information. The last layer is a softmax layer or a sigmoid layer for multi-class classification tasks, which outputs the probability distribution of each class. The last or penultimate layer of a convolutional neural network usually generates a high-dimensional feature vector, which contains comprehensive information about the image. It is a semantic representation of the image and can reflect the core features of the image, such as object category, scene content, etc. The feature vector is used for classification tasks or passed as input to other network modules for further analysis.

[0021] S4: Calculate the product of the transpose of the image understanding feature vector and the text understanding feature vector to obtain a multimodal demand expression matrix; The image understanding feature vector is , whose dimension is 1*m, and the calculation formula is: ; In the formula, is the i-th value of the image feature vector; The text understanding feature vector is , whose dimension is 1*n, and the calculation formula is: ; In the formula, is the jth value of the text feature vector; Understanding feature vectors for images Perform a transposition operation to transform it from a 1*m row vector to an m*1 column vector, and transpose the image feature vector With text feature vector Multiply them together to get an m*n matrix. The calculation formula is: ; Matrix multiplication is calculated as follows: is an m*1 column vector, is a 1*n row vector; Results Matrix The dimension is m*n, and each element represents the correlation between the image and the text in the corresponding dimension; The element calculation formula is: ; Where i=1,...,m and j=1,...,n, indicating that the calculation of each element in the matrix is ​​the element-by-element product of image features and text features.

[0022] Assume that the dimensions of the image feature vector and text feature vector are m=2 and n=3 respectively; ; ; The transposed image feature vector is: ; The matrix multiplication is calculated as: ; The obtained m*n multimodal demand expression matrix is: ; ; S5: Based on the eigenvalues ​​of each position in the multimodal demand expression matrix, each eigenvalue in the matrix is ​​adjusted through a feature correction mechanism; The eigenvalues ​​in the matrix are adjusted through the feature correction mechanism to calculate the importance of text features and image features. The calculation formula is: ; In the formula, is the weight of the i-th feature of the text, reflecting the importance of the text feature. is the i-th text feature. TF-IDF is a weight evaluation method based on text frequency and document frequency, which measures the relative importance of each word. The edge strength in the image is used as the weight of the image feature, and the calculation formula is: ; In the formula, is the weight of the jth feature of the image, reflecting the importance of the image feature. is the intensity corresponding to image feature j, is the maximum intensity of all features in the image; Each element in the multimodal demand expression matrix is ​​corrected by the text feature weight and the image feature weight. Each eigenvalue in the matrix is ​​weighted by the corresponding text and image feature weights. The calculation formula of the corrected matrix element is: ; In the formula, is the element in the original matrix, representing the relationship between text feature i and image feature j, is the corrected eigenvalue, after weighted adjustment, It reflects the combined contribution of text and image features.

[0023] S6: input the corrected multimodal demand expression matrix into a multi-label classifier for processing, and generate a set of classification results through the reasoning of the classifier, which represents the topic labels of the user input request; The corrected multimodal demand expression matrix will be passed to the classifier for further feature mapping. The classifier will fuse and map the input features. For deep learning models, feature mapping is achieved through a multi-layer neural network. The neural network learns the complex relationship between features through multiple hidden layers. The core function of the classifier is to process the input multimodal features and generate the predicted probability of each label. For each label, the classifier will predict a probability value to indicate the relevance of the label to the user request. The sigmoid activation function is used in the last layer to output an independent probability value for each label, indicating whether the label is related to the input request. The calculation formula is: ; In the formula, (·) is the sigmoid function, is the input matrix The i-th type output; According to the output of the classifier, two operations are included: Threshold selection: Compare the probability value of each category with the set threshold. T , then the tag is selected as the topic tag; Top-K selection: If you want to return multiple tags, you can select the top K tags based on the probability value, which indicates the tags most relevant to the user request.

[0024] S7: Generate high-quality text content that meets user needs based on the classification results and the identified topic tags.

[0025] The topic labels obtained according to the classification results are mapped to the corresponding content generation model. For example, each label can correspond to a specific content generation template or rule. The specific meaning of the label will help select or adjust the parameters or structure of the generation model. Through the identified label, the text generation template related to it is selected, or the placeholder in the generation template will be filled according to the label.

[0026] Based on the classification results and the identified topic tags, high-quality text content that meets user needs is generated, and the text generation process is further optimized to ensure that the generated content not only accurately conveys the information required by users, but also meets the requirements of specific scenarios and styles. This process combines multimodal data with user historical behavior and preferences to intelligently adjust the structure, tone, emotional color and expression of the text to improve the relevance, fluency and personalization of the text. At the same time, the system will also provide real-time feedback and optimization based on the quality of the generated text to ensure that the text content has high adaptability and high precision in different application scenarios, thereby truly realizing the creation of high-quality text customized for users and meeting their needs.

[0027] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device.

[0028] Although embodiments of the present invention have been shown and described, it will be appreciated by those skilled in the art that various changes, modifications, substitutions and variations may be made to the embodiments without departing from the principles and spirit of the present invention, and that the scope of the present invention is defined by the appended claims and their equivalents.

Claims

1. A text processing feedback system based on multimodal data, characterized in that: It includes a multimodal data input module, a text semantic feature extraction module, a multi-scale feature extraction module, an image understanding module, a multimodal demand expression matrix generation module, a feature correction module and a text generation optimization module; The multimodal data input module is used to receive multimodal data input by a user, including text, image, voice or video; The text semantic feature extraction module is used to extract semantic feature vectors from the text input by the user; The multi-scale feature extraction module is used to perform multi-scale feature extraction on text semantic features; The image understanding module is used to extract image feature vectors from image data provided by a user; The multimodal demand expression matrix generation module is used to fuse text features and image features to generate a multimodal demand expression matrix; The feature correction module is used to correct the features in the multimodal demand expression matrix; The text generation optimization module generates text content based on the corrected multimodal demand expression matrix and optimizes it according to user feedback.

2. A text processing feedback method based on multimodal data, applied to the text processing feedback system based on multimodal data according to claim 1, characterized in that: The following steps are involved: S1: The text data input by the user is processed through a context encoder containing an embedding layer to obtain semantic feature vectors of multiple texts. Natural language processing technology is used to deeply understand the text and capture the semantic information in it. S2: Arrange multiple semantic feature vectors into a one-dimensional feature vector and process the vector through a multi-scale neighborhood feature extraction module; S3: Process the image data provided by the user through a convolutional neural network to extract the key feature vectors in the image; S4: Calculate the product of the transpose of the image understanding feature vector and the text understanding feature vector to obtain a multimodal demand expression matrix; S5: Based on the eigenvalues ​​of each position in the multimodal demand expression matrix, each eigenvalue in the matrix is ​​adjusted through a feature correction mechanism; S6: input the corrected multimodal demand expression matrix into a multi-label classifier for processing, and generate a set of classification results through the reasoning of the classifier, which represents the topic labels of the user input request; S7: Generate high-quality text content that meets user needs based on the classification results and the identified topic tags.

3. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S1 further includes using a pre-trained word embedding model to convert words or subwords into vector representations of fixed dimensions, and dynamically generating word vectors through the model to capture contextual information. BERT uses bidirectional encoding to capture information in the text from different angles through multiple attention heads. The model selectively focuses on different parts of the input sequence through a self-attention mechanism, integrates information at all positions, generates deep semantic features, performs weighted averaging or pooling on word vectors in sentences or documents, generates text-level semantic representations, and strengthens word embedding through a context encoder to generate word vectors containing contextual information. Multiple word vectors are aggregated to obtain semantic vectors of sentences or paragraphs, and features of multiple layers are extracted through a multi-level encoder.

4. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S2 further includes integrating the semantic feature vectors extracted from multiple texts into a unified representation, sequentially splicing the multiple semantic feature vectors into a long vector, and performing multi-scale processing on the spliced ​​feature vectors to capture semantic information at different levels.

5. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S3 further includes using a convolution layer to extract low-level features in the image, sliding convolution kernels of different sizes on the image to extract features of local areas, each feature map corresponds to a specific pattern in the image, and the convolution neural network includes multiple convolution layers, each layer using convolution kernels of different sizes.

6. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S4 further comprises: The image understanding feature vector is , whose dimension is 1*m, and the calculation formula is: ; In the formula, is the i-th value of the image feature vector; The text understanding feature vector is , whose dimension is 1*n, and the calculation formula is: ; In the formula, is the jth value of the text feature vector; Understanding feature vectors for images Perform a transposition operation to transform it from a 1*m row vector to an m*1 column vector, and transpose the image feature vector With text feature vector Multiply them together to get an m*n matrix. The calculation formula is: ； Matrix multiplication is calculated as follows: is an m*1 column vector, is a 1*n row vector; Results Matrix The dimension is m*n, and each element represents the correlation between the image and the text in the corresponding dimension; The element calculation formula is: ; Where i=1,...,m and j=1,...,n, indicating that the calculation of each element in the matrix is ​​the element-by-element product of image features and text features.

7. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S5 further includes adjusting the eigenvalues ​​in the matrix through a feature correction mechanism, and calculating the importance of text features and image features, and the calculation formula is: ； In the formula, is the weight of the i-th feature of the text, reflecting the importance of the text feature. is the i-th text feature. TF-IDF is a weight evaluation method based on text frequency and document frequency, which measures the relative importance of each word. The edge strength in the image is used as the weight of the image feature, and the calculation formula is: ； In the formula, is the weight of the jth feature of the image, reflecting the importance of the image feature. is the intensity corresponding to image feature j, is the maximum intensity of all features in the image; Each element in the multimodal demand expression matrix is ​​corrected by the text feature weight and the image feature weight. Each eigenvalue in the matrix is ​​weighted by the corresponding text and image feature weights. The calculation formula of the corrected matrix element is: ； In the formula, is the element in the original matrix, representing the relationship between text feature i and image feature j, is the corrected eigenvalue, after weighted adjustment, It reflects the combined contribution of text and image features.

8. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S6 further includes that the corrected multimodal demand expression matrix will be passed to the classifier for further feature mapping. The classifier will fuse and map the input features. For the deep learning model, the feature mapping is implemented by a multi-layer neural network. The neural network learns the complex relationship between features through multiple hidden layers. The core function of the classifier is to process the input multimodal features and generate a predicted probability for each label. For each label, the classifier will predict a probability value to indicate the relevance of the label to the user request. The sigmoid activation function is used in the last layer to output an independent probability value for each label, indicating whether the label is related to the input request. The calculation formula is: ； In the formula, (·) is the sigmoid function, is the input matrix The i-th type output.

9. The text processing feedback method based on multimodal data according to claim 2, characterized in that: The S7 further includes generating text content related to the topic tag through a generation model according to the output of the classifier. During the generation process, the input classification label is used as a condition to guide the text generation model to output text content that meets the user's needs.

Images