Visual language interaction method and system based on multi-modal large model

Abstract

The invention discloses a visual language interaction method and system based on a multi-modal large model, and relates to the technical field of visual language interaction.The method comprises the steps that multi-modal data are collected and preprocessed; a multi-modal large model is constructed, a multi-modal fusion module is additionally arranged in the large model, the module adopts an attention mechanism, the large model can automatically learn the importance degree between different modal data, and effective fusion of multi-modal information is achieved; inputting the preprocessed multi-modal data into a large model, and training and optimizing the large model, so that the large model can better process the multi-modal data; and inputting multi-modal information to the large model, wherein the large model specifically executes the following operations: performing feature extraction on the multi-modal information, fusing features of different modals through a multi-modal fusion module, mining semantic association therein, generating an interactive response according to the fused features, and presenting the interactive response to a user according to a preset output form. The visual language interaction experience in different scenes can be met.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of visual language interaction, in particular to a visual language interaction method and system based on a multi-modal large model. BACKGROUND

[0002] With the development of artificial intelligence technology, visual language interaction has wide application requirements in intelligent customer service, smart home, intelligent driving and many other fields. However, most of the existing visual language interaction methods adopt single modal processing or simple multi-modal splicing methods, which are difficult to comprehensively and accurately understand complex visual and language information. For example, in the intelligent customer service scenario, when the user describes the problem in the form of combination of image and text, the traditional method cannot effectively fuse the image and text information, resulting in inaccurate interaction results; in the intelligent home control, the user controls the device through voice and gesture instructions, and the traditional interaction method is insufficient in the collaborative processing of multi-modal information, affecting the user experience.

[0003] Therefore, there is an urgent need for a more advanced visual language interaction method and system based on a multi-modal large model to improve the accuracy and naturalness of visual language interaction. SUMMARY

[0004] In the current field of visual language interaction, the traditional method has the problems of poor interaction effect and insufficient understanding of complex scenes. Based on this, the present application provides a visual language interaction method and system based on a multi-modal large model.

[0005] In the first aspect, the present application provides a visual language interaction method and system based on a multi-modal large model, which solves the above technical problems by adopting the following technical solutions:

[0006] A visual language interaction method based on a multi-modal large model, comprising the following steps:

[0007] S1, collecting multi-modal data and performing cleaning, labeling and normalization preprocessing operations on the collected multi-modal data;

[0008] S2, constructing a multi-modal large model, adding a multi-modal fusion module to the multi-modal large model, and using an attention mechanism in the multi-modal fusion module to enable the multi-modal large model to automatically learn the importance of different modal data and realize effective fusion of multi-modal information;

[0009] S3, inputting the preprocessed multi-modal data into the multi-modal large model, calculating the loss function between the predicted result and the true label, updating the parameters of the multi-modal large model using the back propagation algorithm, and continuously optimizing the multi-modal large model to make it better process multi-modal data;

[0010] S4, input the multi-modal information input by the user into the optimized multi-modal large model, and the multi-modal large model specifically performs the following operations: first, feature extraction is performed on the multi-modal information, then different modal features are fused through a multi-modal fusion module to mine semantic associations therein, and finally an interactive response is generated according to the fused features;

[0011] S5, according to the specific type of the interactive response, the generated interactive response is presented to the user in a preset output form.

[0012] Optionally, the step S1 specifically includes:

[0013] Collection and preprocessing of image data: image data covering various scenes and categories are collected from public data sets or web crawlers; first, remove ambiguous, distorted or irrelevant images through cleaning to ensure data quality, then perform denoising on the retained images to eliminate noise interference, and finally crop the images according to the interactive requirements to retain the key information area; label the categories and attribute information of the images, and establish associated annotations with text data and speech data to ensure semantic consistency; scale the images to a uniform size, convert them to a standardized format, and adjust the brightness and contrast parameters to eliminate the influence of size, format and visual feature differences on the input of the multi-modal large model;

[0014] Collection and preprocessing of text data: collect text data related to images; first, remove stop words, special symbols and redundant information through cleaning, correct misspelled words or grammatical errors, then perform word segmentation to divide sentences into words, and complete part-of-speech tagging to identify word parts-of-speech and lay the foundation for semantic analysis; label the semantics of the text, and establish cross-modal associated annotations to ensure semantic consistency; encode the text to convert the text after part-of-speech tagging into a vector form that can be processed by the multi-modal large model, providing adaptive input for the training and inference of the multi-modal large model;

[0015] Collection and preprocessing of speech data: collect user speech data; first, perform noise reduction and silence segment truncation through cleaning, and unify the speech sampling rate and duration, then convert the speech into text form through speech recognition technology; based on the converted text, establish associated annotations with image data and text data to ensure cross-modal semantic consistency; encode the text data converted from the speech to ensure that its feature dimension is adapted to that of the image data and text data, and finally provide a unified input format for the training and inference of the multi-modal large model.

[0016] Optionally, the step S4 specifically includes:

[0017] The multi-modal large model performs convolution operation on the image through an image feature extraction network to extract the feature vector of the image, and performs word embedding and encoding processing on the text through a text feature extractor to generate the feature vector of the text;

[0018] The image feature vector and the text feature vector are input into a multi-modal fusion module, the multi-modal fusion module calculates the correlation weight between different modal features by using an attention mechanism, the features are weighted and fused based on the weight, and a fusion feature vector containing cross-modal semantic correlation is obtained;

[0019] The fusion feature vector is input into the output layer of the multi-modal large model, and after full connection layer and subsequent decoding processing, an interactive response meeting the task requirement is generated.

[0020] Optionally, step S5 is performed, and the interactive response includes three types of text, image and voice;

[0021] For a text type interactive response, the text content is directly displayed in a specified area of the interactive interface;

[0022] For an image type interactive response, the generated or loaded image is rendered and displayed to the user in a preset format in the interface;

[0023] For a voice type interactive response, the text is converted into a voice signal through a voice synthesis technology, and then the voice signal is played through an audio output device.

[0024] Preferably, step S2 is performed, and the multi-modal large model is constructed based on a Transformer architecture;

[0025] Step S3 is performed, and the preprocessed multi-modal data is input into the multi-modal large model according to a preset batch.

[0026] In a second aspect, the present application provides a visual language interaction method and system based on a multi-modal large model, which solves the above technical problems by adopting the following technical solutions:

[0027] A visual language interaction system based on a multi-modal large model includes:

[0028] A data collection and preprocessing module is configured to collect multi-modal data and perform cleaning, labeling and normalization preprocessing operations on the collected multi-modal data;

[0029] A multi-modal large model construction module is configured to construct a multi-modal large model, and a multi-modal fusion module is added to the multi-modal large model, the multi-modal fusion module adopts an attention mechanism, which enables the multi-modal large model to automatically learn the importance of different modal data and effectively fuse multi-modal information;

[0030] A multi-modal large model optimization module is configured to input the preprocessed multi-modal data into the multi-modal large model, update the parameters of the multi-modal large model by calculating the loss function between the predicted result and the real label, and continuously optimize the multi-modal large model so that it can better process multi-modal data.

[0031] The multi-modal large model is used to firstly extract features of multi-modal information, then fuse features of different modalities through a multi-modal fusion module to mine semantic correlations therein, and finally generate an interactive response according to the fused features;

[0032] The visualization module provides an interactive interface for a user to input multi-modal information, inputs the multi-modal information input by the user into the completed multi-modal large model, and displays the generated interactive response in the interactive interface according to a preset output form according to a specific type of the interactive response.

[0033] Optionally, the data collection and preprocessing module specifically includes:

[0034] The image collection and preprocessing unit is responsible for performing the following operations: collecting image data covering various scenes and categories from public data sets or web crawlers; removing ambiguous, distorted or irrelevant images through cleaning to ensure data quality, then performing denoising processing on the retained images to eliminate noise interference, and finally cropping the images according to interactive requirements to retain key information areas; labeling categories and attribute information for the images, and establishing associated annotations with text data and speech data to ensure semantic consistency; scaling the images to a uniform size, converting them to a standardized format, and adjusting brightness and contrast parameters to eliminate the influence of size, format and visual feature differences on the input of the multi-modal large model;

[0035] The text collection and preprocessing unit is responsible for performing the following operations: collecting text data related to images; removing stop words, special symbols and redundant information through cleaning to correct misspelled words or grammatical errors, then performing word segmentation to divide sentences into words and completing part-of-speech tagging to identify word parts-of-speech and lay a foundation for semantic analysis; labeling semantic categories for the text, and establishing cross-modal associated annotations to ensure semantic consistency; encoding the text to convert the text after part-of-speech tagging into a vector form that can be processed by the multi-modal large model, providing adaptive input for the training and inference of the multi-modal large model;

[0036] The speech collection and preprocessing unit is responsible for performing the following operations: collecting speech data of a user; performing noise reduction and silent segment truncation processing through cleaning to unify speech sampling rate and duration, then converting the speech into text form through speech recognition technology; establishing associated annotations with image data and text data based on the converted text to ensure cross-modal semantic consistency; encoding the text data converted from the speech to ensure that it is adapted to the feature dimension of the image data and the text data, and finally providing a unified input format for the training and inference of the multi-modal large model.

[0037] Optionally, the multi-modal large model specifically performs the following operations:

[0038] The multi-modal large model performs convolution operation on the image through an image feature extraction network to extract a feature vector of the image, and performs word embedding and encoding processing on the text through a text feature extractor to generate a feature vector of the text;

[0039] The image feature vector and the text feature vector are input into a multi-modal fusion module, the multi-modal fusion module calculates the correlation weight between different modal features by using an attention mechanism, weights the features based on the weight, and obtains a fusion feature vector containing cross-modal semantic correlation;

[0040] The fusion feature vector is input into an output layer of the multi-modal large model, and after full connection layer and subsequent decoding processing, an interactive response meeting the task requirement is generated.

[0041] Optionally, the interactive response includes three types of text, image and voice;

[0042] For the text type interactive response, the visualization module directly displays the text content in a specified area of the interactive interface;

[0043] For the image type interactive response, the visualization module renders and displays the generated or loaded image to the user in a preset format in the interface;

[0044] For the voice type interactive response, the visualization module converts the text into a voice signal through a speech synthesis technology, and then plays it through an audio output device.

[0045] Preferably, the multi-modal large model construction module constructs the multi-modal large model based on the Transformer architecture;

[0046] The multi-modal large model optimization module inputs the preprocessed multi-modal data into the multi-modal large model according to a preset batch.

[0047] Compared with the prior art, the visual language interaction method and system based on the multi-modal large model has the following beneficial effects:

[0048] 1. The present application first collects and preprocesses image, text, voice and other multi-modal data, trains a multi-modal large model, when the user inputs multi-modal information, uses the trained model to process the interaction and output the result, which can significantly improve the performance and user experience of visual language interaction, and solves the problem of insufficient multi-modal information processing capability and unsatisfactory interaction effect of traditional visual language interaction method;

[0049] 2. This invention, by fusing multimodal data and processing it using a large multimodal model, can more comprehensively and deeply understand the user's intent and improve the accuracy of visual language interaction. The designed multimodal fusion module can effectively integrate information from different modalities, enhancing the ability of the large multimodal model to handle complex scenarios. It has good versatility and scalability, and can be applied to multiple fields to improve the visual language interaction experience in different scenarios. Attached Figure Description

[0050] Appendix Figure 1 This is a flowchart of the method described in Embodiment 1 of the present invention;

[0051] Appendix Figure 2 This is a detailed flowchart of the method described in Embodiment 1 of the present invention;

[0052] Appendix Figure 3 This is a module connection block diagram of Embodiment 2 of the present invention. Detailed Implementation

[0053] To make the technical solution, the technical problem solved, and the technical effect of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with specific embodiments.

[0054] Example 1:

[0055] Reference Appendix Figure 1 and 2 This embodiment proposes a visual language interaction method based on a multimodal large model, which includes the following steps:

[0056] S1. Collect multimodal data and perform preprocessing operations such as cleaning, labeling, and normalization on the collected multimodal data. This process specifically includes:

[0057] (A) Image Data Collection and Preprocessing: Collect image data covering various scenes and categories from public datasets or web crawlers; first, clean and remove blurry, distorted, or irrelevant images to ensure data quality; then, denoise the remaining images to eliminate noise interference; finally, crop the images according to interaction requirements, retaining key information areas; label the images with category and attribute information, and establish association annotations with text and voice data to ensure semantic consistency; scale the images to a uniform size and convert them to a standardized format, while adjusting brightness and contrast parameters to eliminate the impact of size, format, and visual feature differences on the input of multimodal large models;

[0058] (B) Collection and preprocessing of text data: collect text data related to images; first remove stop words, special symbols and redundant information by cleaning, correct spelling or grammar errors, then perform word segmentation operation to divide sentences into words, and complete part-of-speech tagging to identify word parts-of-speech, laying a foundation for semantic analysis; label the semantics of the text, and at the same time establish cross-modal correlation labeling to ensure semantic consistency; encode the text to convert the text after part-of-speech tagging into a vector form that can be processed by the multi-modal large model, providing adaptive input for the training and inference of the multi-modal large model;

[0059] (C) Collection and preprocessing of speech data: collect user speech data; first, clean up and perform noise reduction, truncate silent segments, and unify the speech sampling rate and duration; then convert the speech into text format through speech recognition technology; based on the converted text, establish associated labeling with image data and text data to ensure cross-modal semantic consistency; encode the text data converted from speech to ensure that its feature dimension is adapted to that of image data and text data, and finally provide a unified input format for the training and inference of the multi-modal large model.

[0060] S2, constructing a multi-modal large model based on a Transformer architecture, adding a multi-modal fusion module to the multi-modal large model, the multi-modal fusion module using an attention mechanism that can enable the multi-modal large model to automatically learn the importance of different modal data, and effectively fuse multi-modal information.

[0061] S3, inputting the preprocessed multi-modal data into the multi-modal large model according to a preset batch, updating the parameters of the multi-modal large model using a backpropagation algorithm by calculating the loss function between the predicted results and the true labels, and continuously optimizing the multi-modal large model to make it better handle multi-modal data.

[0062] S4, inputting the multi-modal information input by the user into the optimized multi-modal large model, the multi-modal large model specifically performing the following operations: first, extracting features from the multi-modal information, then fusing the features of different modalities through the multi-modal fusion module to mine the semantic associations therein, and finally generating an interactive response based on the fused features.

[0063] This process specifically includes:

[0064] The multi-modal large model performs convolution operation on the image through the image feature extraction network to extract the feature vector of the image, and performs word embedding and encoding processing on the text through the text feature extractor to generate the feature vector of the text;

[0065] The image feature vector and the text feature vector are input into a multi-modal fusion module. The multi-modal fusion module calculates the correlation weight between different modal features by using an attention mechanism, weights the features based on the weight, and obtains a fusion feature vector containing cross-modal semantic correlation.

[0066] The fusion feature vector is input into an output layer of a multi-modal large model. After full connection and subsequent decoding processing, an interactive response meeting the task requirement is generated.

[0067] S5, according to the specific type of the interactive response, the generated interactive response is presented to the user in a preset output form.

[0068] The interactive response includes text, image and voice;

[0069] For a text type interactive response, the text content is directly displayed in a specified area of an interactive interface;

[0070] For an image type interactive response, the generated or loaded image is rendered and displayed to the user in a preset format (such as JPG, PNG) in the interface;

[0071] For a voice type interactive response, the text is converted into a voice signal through a speech synthesis (TTS) technology, and then the voice signal is played through an audio output device (such as a loudspeaker or a headset).

[0072] Embodiment two:

[0073] Reference is made to the accompanying drawings Figure 3 The embodiment provides a visual language interaction system based on a multi-modal large model, which comprises:

[0074] A data collection and preprocessing module is configured to collect multi-modal data and perform preprocessing operations such as cleaning, labeling and normalization on the collected multi-modal data;

[0075] A multi-modal large model construction module is configured to construct a multi-modal large model based on a Transformer architecture, and add a multi-modal fusion module to the multi-modal large model. The multi-modal fusion module adopts an attention mechanism, which can enable the multi-modal large model to automatically learn the importance of different modal data and effectively fuse multi-modal information.

[0076] A multi-modal large model optimization module is configured to input the preprocessed multi-modal data into the multi-modal large model in batches according to a preset batch, calculate the loss function between the prediction result and the real label, update the parameters of the multi-modal large model by using a back propagation algorithm, and continuously optimize the multi-modal large model so that it can better process multi-modal data.

[0077] The multi-modal large model is used for firstly extracting features of multi-modal information, then fusing features of different modalities through a multi-modal fusion module to mine semantic correlations therein, and finally generating an interactive response according to the fused features;

[0078] The visualization module provides an interactive interface for a user to input multi-modal information, inputs the multi-modal information input by the user into the completed multi-modal large model, and displays the generated interactive response in the interactive interface according to a preset output form according to a specific type of the interactive response.

[0079] In this embodiment, the data collection and preprocessing module specifically includes:

[0080] The image collection and preprocessing unit is responsible for performing the following operations: collecting image data covering various scenes and categories from public data sets or web crawlers; removing ambiguous, distorted or irrelevant images through cleaning to ensure data quality, then performing denoising processing on the retained images to eliminate noise interference, and finally cropping the images according to interactive requirements to retain key information areas; labeling categories and attribute information for the images, and establishing associated annotations with text data and speech data to ensure semantic consistency; scaling the images to a uniform size, converting them to a standardized format, and adjusting brightness and contrast parameters to eliminate the influence of size, format and visual feature differences on the input of the multi-modal large model;

[0081] The text collection and preprocessing unit is responsible for performing the following operations: collecting text data related to images; removing stop words, special symbols and redundant information through cleaning to correct misspelled words or grammatical errors, then performing word segmentation to divide sentences into words and completing part-of-speech tagging to identify word parts-of-speech and lay a foundation for semantic analysis; labeling semantic categories for the text, and establishing cross-modal associated annotations to ensure semantic consistency; encoding the text to convert the text after part-of-speech tagging into a vector form that can be processed by the multi-modal large model, providing adaptive input for the training and inference of the multi-modal large model;

[0082] The speech collection and preprocessing unit is responsible for performing the following operations: collecting speech data of a user; performing noise reduction and silent segment truncation processing through cleaning to unify speech sampling rate and duration, then converting the speech into text form through speech recognition technology; establishing associated annotations with image data and text data based on the converted text to ensure cross-modal semantic consistency; encoding the text data converted from the speech to ensure that the feature dimensions of the text data are adapted to those of the image data and the text data, and finally providing a unified input format for the training and inference of the multi-modal large model.

[0083] In this embodiment, the multi-modal large model specifically performs the following operations:

[0084] The multi-modal large model performs convolution operation on the image through an image feature extraction network to extract a feature vector of the image, and performs word embedding and encoding processing on the text through a text feature extractor to generate a feature vector of the text;

[0085] The image feature vector and the text feature vector are input into a multi-modal fusion module, the multi-modal fusion module calculates correlation weights between different modal features by using an attention mechanism, and performs weighted fusion on the features based on the weights to obtain a fusion feature vector containing cross-modal semantic correlation;

[0086] The fusion feature vector is input into an output layer of the multi-modal large model, and after full connection layer and subsequent decoding processing, an interactive response meeting the task requirement is generated.

[0087] In this embodiment, the interactive responses include three types of text, image and voice;

[0088] For the text type interactive response, the visualization module directly displays the text content in a specified area of the interactive interface;

[0089] For the image type interactive response, the visualization module renders and displays the generated or loaded image to the user in the interface according to a preset format (such as JPG, PNG);

[0090] For the voice type interactive response, the visualization module converts the text into a voice signal through a text-to-speech (TTS) technology, and then plays the voice signal through an audio output device (such as a loudspeaker or a headset).

[0091] As can be seen from the above, the visual language interaction method and system based on the multi-modal large model can collect and preprocess various modal data such as images, texts and voices, train a multi-modal large model, and when multi-modal information is input by a user, the trained model is used for interactive processing and output of results, which can significantly improve the performance and user experience of visual language interaction, and solve the problems of insufficient processing capacity of multi-modal information and unsatisfactory interactive effect of the traditional visual language interaction method.

[0092] The above application specific examples have described the principles and implementation modes of the present application in detail, and these examples are only used to help understand the core technical content of the present application. Based on the above specific embodiments of the present application, any improvement and modification of the present application made by those skilled in the art without departing from the principles of the present application shall fall within the patent protection scope of the present application.

Claims

1. A visual language interaction method based on a multimodal large model, characterized in that, Includes the following steps: S1. Collect multimodal data and perform preprocessing operations such as cleaning, labeling, and normalization on the collected multimodal data; S2. Construct a multimodal large model and add a multimodal fusion module to the multimodal large model. The multimodal fusion module adopts an attention mechanism, which enables the multimodal large model to automatically learn the importance between different modal data and realize the effective fusion of multimodal information. S3. Input the preprocessed multimodal data into the multimodal large model, calculate the loss function between the prediction results and the true labels, and use the backpropagation algorithm to update the parameters of the multimodal large model, continuously optimizing the multimodal large model so that it can better handle multimodal data. S4. Input the multimodal information input by the user into the optimized multimodal large model. The multimodal large model performs the following operations: First, extract features from the multimodal information; then, fuse the features of different modalities through the multimodal fusion module to mine the semantic associations; finally, generate an interactive response based on the fused features. S5. Based on the specific type of the interactive response, present the generated interactive response to the user in a preset output format.

2. The visual language interaction method based on a multimodal large model according to claim 1, characterized in that, Step S1 specifically includes: Image data collection and preprocessing: Collect image data covering various scenes and categories from public datasets or web crawlers; first, clean and remove blurry, distorted, or irrelevant images to ensure data quality; then, denoise the remaining images to eliminate noise interference; finally, crop the images according to interaction requirements, retaining key information areas; label the images with category and attribute information, and establish association annotations with text and voice data to ensure semantic consistency; scale the images to a uniform size and convert them to a standardized format, while adjusting brightness and contrast parameters to eliminate the impact of size, format, and visual feature differences on the input of multimodal large models; Text data collection and preprocessing: Collect text data related to images; first, remove stop words, special symbols, and redundant information through cleaning, correct typos or grammatical errors, then perform word segmentation to divide sentences into words, and complete part-of-speech tagging to identify word parts of speech, laying the foundation for semantic analysis; label the text with semantic categories, and establish cross-modal association tags to ensure semantic consistency; encode the text, converting the part-of-speech-tagged text into a vector form that can be processed by multimodal large-scale models, providing adapted input for the training and inference of multimodal large-scale models; Voice data collection and preprocessing: Collect user voice data; first, perform noise reduction and truncation of silent segments through cleaning, and unify the voice sampling rate and duration; then, convert the voice into text form through speech recognition technology; establish association annotations between the converted text and image data and text data to ensure cross-modal semantic consistency; encode the voice-converted text data to ensure that it is compatible with the feature dimensions of image data and text data, and finally provide a unified input format for the training and inference of multimodal large models.

3. The visual language interaction method based on a multimodal large model according to claim 2, characterized in that, Step S4 specifically includes: The multimodal large model performs convolution operations on images through an image feature extraction network to extract the feature vectors of the images, and performs word embedding and encoding processing on the text through a text feature extractor to generate the feature vectors of the text; Image feature vectors and text feature vectors are input into the multimodal fusion module. The multimodal fusion module uses an attention mechanism to calculate the association weights between features of different modalities. Based on the weights, the features are weighted and fused to obtain a fused feature vector containing cross-modal semantic associations. The output layer of a multimodal large model is fused with feature vector inputs, and after passing through a fully connected layer and subsequent decoding processing, an interactive response that meets the task requirements is generated.

4. The visual language interaction method based on a multimodal large model according to claim 3, characterized in that, Execute step S5; the interactive response includes three types: text, image, and voice. For text-based interactive responses, the text content is directly displayed in the specified area of ​​the interactive interface; For image-based interactive responses, the generated or loaded image will be rendered and displayed to the user in the interface according to a preset format; For voice-based interactive responses, text is converted into speech signals using speech synthesis technology, and then broadcast through an audio output device.

5. A visual language interaction method based on a multimodal large model according to any one of claims 1-4, characterized in that, Execute step S2 to build a multimodal large model based on the Transformer architecture; Execute step S3 to input the preprocessed multimodal data into the multimodal large model according to the preset batch.

6. A visual language interaction system based on a multimodal large model, characterized in that, It includes: The data collection and preprocessing module is used to collect multimodal data and perform preprocessing operations such as cleaning, labeling, and normalization on the collected multimodal data. The multimodal large model construction module is used to build multimodal large models. A multimodal fusion module is added to the multimodal large model. The multimodal fusion module adopts an attention mechanism, which enables the multimodal large model to automatically learn the importance between different modal data and realize the effective fusion of multimodal information. The multimodal large model optimization module is used to input preprocessed multimodal data into the multimodal large model, calculate the loss function between the predicted results and the true labels, and use the backpropagation algorithm to update the parameters of the multimodal large model, continuously optimizing the multimodal large model so that it can better handle multimodal data; The multimodal large model is used to first extract features from multimodal information, then fuse the features of different modalities through the multimodal fusion module, mine the semantic associations, and finally generate interactive responses based on the fused features. The visualization module provides an interactive interface for users to input multimodal information. It also inputs the user-input multimodal information into the optimized multimodal large model and displays the generated interactive response on the interactive interface according to the specific type of the interactive response and a preset output format.

7. A visual language interaction system based on a multimodal large model according to claim 6, characterized in that, The data collection and preprocessing module specifically includes: The image collection and preprocessing unit is responsible for performing the following operations: collecting image data covering various scenes and categories from public datasets or web crawlers; first, cleaning and removing blurry, distorted, or irrelevant images to ensure data quality; then, denoising the remaining images to eliminate noise interference; finally, cropping the images according to interaction requirements while retaining key information areas; labeling images with category and attribute information, and establishing association labels with text and voice data to ensure semantic consistency; scaling images to a uniform size and converting them to a standardized format, while adjusting brightness and contrast parameters to eliminate the impact of size, format, and visual feature differences on the input of multimodal large models; The text collection and preprocessing unit is responsible for performing the following operations: collecting text data related to images; first, cleaning and removing stop words, special symbols, and redundant information, correcting typos or grammatical errors, then performing word segmentation to divide sentences into words and completing part-of-speech tagging to identify word parts of speech, laying the foundation for semantic analysis; labeling the text with semantic categories, while establishing cross-modal association tags to ensure semantic consistency; and encoding the text, converting the tagged text into a vector form that can be processed by the multimodal large model, providing adapted input for the training and inference of the multimodal large model. The speech collection and preprocessing unit is responsible for performing the following operations: collecting users' speech data; first, performing noise reduction and truncation of silent segments through cleaning, and unifying the speech sampling rate and duration, and then converting the speech into text form through speech recognition technology; establishing association annotations between the converted text and image data and text data to ensure cross-modal semantic consistency; encoding the speech-converted text data to ensure that it matches the feature dimensions of image data and text data, and finally providing a unified input format for the training and inference of multimodal large models.

8. A visual language interaction system based on a multimodal large model according to claim 7, characterized in that, The multimodal large model specifically performs the following operations: The multimodal large model performs convolution operations on images through an image feature extraction network to extract the feature vectors of the images, and performs word embedding and encoding processing on the text through a text feature extractor to generate the feature vectors of the text; Image feature vectors and text feature vectors are input into the multimodal fusion module. The multimodal fusion module uses an attention mechanism to calculate the association weights between features of different modalities. Based on the weights, the features are weighted and fused to obtain a fused feature vector containing cross-modal semantic associations. The output layer of a multimodal large model is fused with feature vector inputs, and after passing through a fully connected layer and subsequent decoding processing, an interactive response that meets the task requirements is generated.

9. A visual language interaction system based on a multimodal large model according to claim 8, characterized in that, The interactive responses include three types: text, image, and voice. For text-based interactive responses, the visualization module directly displays the text content in a designated area of ​​the interactive interface; For image-based interactive responses, the visualization module will render and display the generated or loaded images to the user in a preset format on the interface. For voice-based interactive responses, the visualization module uses speech synthesis technology to convert text into speech signals, which are then broadcast through an audio output device.

10. A visual language interaction system based on a multimodal large model according to any one of claims 6-9, characterized in that, The multimodal large model construction module constructs multimodal large models based on the Transformer architecture; The multimodal large model optimization module inputs the preprocessed multimodal data into the multimodal large model according to a preset batch.

Images