Multimodal recognition method, device, electronic device and storage medium based on large language model

Abstract

The embodiments of the present application disclose a multimodal recognition method, device, electronic device and storage medium based on a large language model, which belongs to the field of artificial intelligence information processing technology, including collecting text data and different types of modal data, preprocessing the data, combining the text data with the modal data, using the large language model to extract semantic representation features of the text data, and extracting features from the modal data, performing multi-stage multimodal fusion, selecting a multimodal recognition task model architecture, constructing a multimodal recognition model, designing a loss function, using the loss function and an optimizer to train the multimodal recognition model, using a validation set and a test set to evaluate the performance of the multimodal recognition model, adjusting the model hyperparameters and architecture, and optimizing the model. The embodiments of the present application can realize intelligent communication while also performing visual interaction with users to meet the user's usage needs.

Description

Technical Field

[0001] The present application belongs to the field of artificial intelligence information processing technology, and specifically, relates to a multimodal recognition method, device, electronic device and storage medium based on a large language model. Background Art

[0002] A large language model refers to a language model with large-scale parameters. It has powerful language understanding and generation capabilities. It is trained through deep learning technology and uses a large amount of text data to generate coherent, logical, and high-quality text content.

[0003] Multimodal recognition is the process of integrating multiple modal information such as computer vision and natural language processing, associating different modal information, realizing the understanding and analysis of multiple media data, and providing comprehensive and accurate understanding and analysis results.

[0004] The amount and types of multimodal data are constantly increasing. Existing multimodal recognition models find it difficult to discover potential semantic associations between data of different modalities. When conducting question-and-answer sessions, they lack the ability to understand and answer semantic questions raised by users and can only provide simple preset information. They are unable to achieve intelligent communication and cannot meet the personalized needs of users. At the same time, they lack the ability to interact visually with users and the ability to understand and reason about objects in pictures, and cannot meet users' image usage needs. Summary of the invention

[0005] In order to solve the above problems and technical defects, the embodiment of the present application adopts the following technical solution, a multimodal recognition method based on a large language model, comprising the following steps:

[0006] Step 1: collect text data and different types of modal data, pre-process the data, and combine the text data with the modal data to make them consistent in format and representation;

[0007] Step 2: Use a large language model to extract semantic representation features of text data, and perform feature extraction on modal data to make the feature representations of different types of data unified in the same space;

[0008] Step 3: Perform multi-stage multi-modal fusion to fuse the features of different types of data;

[0009] Step 4: Select the multimodal recognition task model architecture, and then build a multimodal recognition model by combining the output results of the large language model and the characteristics of the modal data;

[0010] Step 5. Design a loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

[0011] Preferably, in step 3, multi-stage multi-modal fusion is performed in five stages:

[0012] In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector;

[0013] In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network;

[0014] In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight.

[0015] In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space;

[0016] In the fifth stage, the fusion results of the first and second stages are integrated, and then the integrated results are adjusted and optimized step by step according to the fusion results of the third and fourth stages.

[0017] Preferably, the multimodal recognition model in step 4 needs to determine the specific type of the multimodal recognition task, and design appropriate task modules according to the actual situation, including: image generation task module, question answering task module, image analysis task module, video generation task module;

[0018] The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators;

[0019] The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier;

[0020] The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier;

[0021] The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

[0022] Preferably, when designing the loss function in step 5, it is necessary to optimize according to the information of multiple modal data and task requirements, while considering the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

[0023] A multimodal recognition device based on a large language model, comprising:

[0024] The data collection and preprocessing module is used to collect text data and different types of modal data, preprocess the data, and combine and unify the text data with the modal data so that the text data and the modal data are consistent in format and representation;

[0025] The feature extraction module is used to extract the semantic representation features of text data using a large language model and perform feature extraction on modal data so that the feature representations of different types of data are unified in the same space;

[0026] Feature fusion module, used to perform multi-stage multi-modal fusion and fuse the features of different types of data;

[0027] The model building module selects the multimodal recognition task model architecture, and then combines the output results of the large language model and the characteristics of the modal data to build a multimodal recognition model;

[0028] Train the optimization module, design the loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

[0029] Preferably, the feature fusion module performs multi-stage multi-modal fusion in five stages:

[0030] In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector;

[0031] In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network;

[0032] In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight.

[0033] In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space;

[0034] In the fifth stage, the fusion results of the first and second stages are integrated, and then the integrated results are adjusted and optimized step by step according to the fusion results of the third and fourth stages.

[0035] Preferably, the multimodal recognition model in the model building module needs to determine the specific type of the multimodal recognition task, and design appropriate task modules according to actual conditions, including: image generation task module, question answering task module, image analysis task module, and video generation task module;

[0036] The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators;

[0037] The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier;

[0038] The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier;

[0039] The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

[0040] Preferably, when designing the loss function, the training optimization module needs to optimize according to the information of multiple modal data and task requirements, while considering the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

[0041] An electronic device comprises a memory, a processor and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, the steps of the multimodal recognition method based on a large language model as described above are implemented.

[0042] A computer-readable storage medium stores a computer program, which, when executed by a processor, implements the steps of the multimodal recognition method based on a large language model as described above.

[0043] Compared with the prior art, the beneficial effects of the embodiments of the present application are:

[0044] This application extracts semantic representation features of text data through a large language model, then extracts features from modal data, performs multi-stage multimodal fusion, fuses features of different types of data, and combines the output results of the large language model with the features of the modal data. The generated multimodal recognition model can not only meet the user's text question and answer tasks, but also generate images and video clips according to the user's text requirements. At the same time, it analyzes the user's input image and outputs the content expression expressed by the image, realizing intelligent communication while also interacting visually with the user to meet the user's usage needs. BRIEF DESCRIPTION OF THE DRAWINGS

[0045] In the attached picture:

[0046] Figure 1 A schematic diagram of the method steps of an embodiment of the present application;

[0047] Figure 2 A schematic diagram of a method flow of an embodiment of the present application;

[0048] Figure 3 Schematic diagram of the device structure of an embodiment of the present application. DETAILED DESCRIPTION

[0049] In order to make the purpose, technical solutions and advantages of the embodiments of the present application clearer, the technical solutions in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all of the embodiments. The components of the embodiments of the present application generally described and shown in the drawings here can be arranged and designed in various different configurations.

[0050] Example 1

[0051] like Figure 1 As shown in the figure, the multimodal recognition method based on the large language model has the following specific process:

[0052] Collect text data and different types of modal data, preprocess the data, and combine the text data with the modal data to make them consistent in format and representation.

[0053] The preprocessing of text data is to remove special characters and punctuation marks, unify the case and remove stop words, tokenize the text data, and convert each word into its index in the vocabulary or convert it into a dense vector representation using word embedding technology.

[0054] Modal data preprocessing is performed according to specific circumstances, including scaling, cropping and normalizing image data, and performing time-frequency conversion on audio data.

[0055] The steps to unify text data with modal data include:

[0056] S01. Use word embedding technology to represent text data as a vector of fixed length, and represent modal data as a vector or matrix of fixed length, so that text data and modal data are represented in a unified format;

[0057] S02. Associating the text data with other modal data using the same identifier or index to ensure the correspondence between the text data and the other modal data. If the modal data is a multimodal data set, the correspondence between the text data and the modal data can be obtained according to the metadata file or database of the multimodal data set, and data association and alignment are performed according to these correspondences.

[0058] S03, perform synonym replacement, random word deletion and other operations on text data, perform random rotation, translation or noise addition and other operations on modal data, and enhance the data to increase the diversity and richness of training samples;

[0059] S04. Divide the data sets of text data and modal data into a training set, a validation set, and a test set, so that each set contains the correspondence between the text data and the modal data, and the formats and representations are consistent.

[0060] A large language model is used to extract semantic representation features of text data, and feature extraction is performed on modal data to make the feature representations of different types of data unified in the same space.

[0061] Use word embeddings to extract text features, use convolutional neural networks (CNN) to extract image features, use acoustic features to extract audio features, etc. Use a large language model to extract the semantic representation of the text from the text data, so that the semantic representation can be fused with the features of the modal data. When performing feature fusion, combine the output of the large language model with the features of the modal data to obtain richer and more comprehensive information.

[0062] Perform multi-stage multimodal fusion to fuse the features of different types of data.

[0063] In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector;

[0064] In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network;

[0065] In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight.

[0066] In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space;

[0067] In the fifth stage, the fusion results of the first and second stages are integrated, and then the integrated results are adjusted and optimized step by step according to the fusion results of the third and fourth stages.

[0068] Select the multimodal recognition task model architecture, and then build a multimodal recognition model by combining the output results of the large language model and the characteristics of the modal data.

[0069] The model architecture of the multimodal recognition task includes: a multi-input model that can accept multiple modal inputs; a multi-branch model that can build independent feature extraction branches for different modalities; and a cross-modal model that can simultaneously process information from different modal data and learn the association between information.

[0070] In the multimodal recognition model, it is necessary to determine the specific type of multimodal recognition tasks and add appropriate task modules according to the actual situation, including: image generation task module, question-answering task module, image analysis task module, and video generation task module;

[0071] The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators;

[0072] The language feature extractor uses a recurrent neural network (RNN) or a Transformer structure to extract language features; the visual feature extractor uses a convolutional neural network (CNN) to extract image features; and the image generator uses a recurrent neural network (RNN) or a Transformer structure to generate images described in text.

[0073] The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier;

[0074] The language feature extractor uses a recurrent neural network (RNN) or a Transformer structure to extract language features from the question text; the text classifier inputs the associated features into the classifier and outputs the category of the answer to the question or generates an answer.

[0075] The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier;

[0076] The visual feature extractor uses a convolutional neural network (CNN) to extract image features; the image classifier inputs the extracted image features and outputs the content expressed by the image.

[0077] The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

[0078] The language feature extractor uses a recurrent neural network (RNN) or a Transformer structure to extract language features; the visual feature extractor uses a convolutional neural network (CNN) to extract image features; the image generator uses a recurrent neural network (RNN) or a Transformer structure to generate video image frames with text descriptions; the time series model inputs the generated video image frame features into a recurrent neural network (RNN) or a Transformer to model the time series information of the video; the video generator uses an RNN or a Transformer structure to generate video clips.

[0079] Design loss functions, use loss functions and optimizers to train multimodal recognition models, use validation sets and test sets to evaluate the performance of multimodal recognition models, evaluate the performance of multimodal recognition models on multiple data types, and adjust model hyperparameters and architecture based on the evaluation results to optimize the model.

[0080] When designing the loss function, it is necessary to optimize it according to the information of multiple modal data and task requirements, while considering the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

[0081] Example 2

[0082] After the multimodal recognition model training and optimization are completed, the multimodal recognition model needs to be deployed and applied. The steps are as follows:

[0083] Choose a suitable API framework, such as Flask, Django, etc., define API endpoints, and determine the functions and interfaces that need to be provided, such as uploading multimodal data, performing recognition operations, etc.;

[0084] According to the designed API endpoint, write the corresponding API function code, which can process API requests and call the multimodal recognition model for processing;

[0085] Integrate the multimodal recognition system and integrate the multimodal recognition model into the API service to ensure that the multimodal recognition model can call, receive and process API requests in the API service;

[0086] Perform data transmission in the API service, receive uploaded multimodal data, pass the data to the multimodal recognition model for processing, and return the recognition results to the client;

[0087] Test the API service in the test environment to ensure that the API function runs normally and can correctly call the multimodal recognition model for recognition operations;

[0088] Deploy the API service to the working environment, monitor the operation of the API service in the working environment to ensure its stability and availability, handle abnormal situations in a timely manner, and perform regular system maintenance and updates.

[0089] Example 3

[0090] A multimodal recognition device based on a large language model, comprising:

[0091] The data collection and preprocessing module is used to collect text data and different types of modal data, preprocess the data, and combine and unify the text data with the modal data to make the text data and the modal data consistent in format and representation.

[0092] The feature extraction module is used to extract the semantic representation features of text data using a large language model and perform feature extraction on modal data so that the feature representations of different types of data are unified in the same space.

[0093] The feature fusion module is used to perform multi-stage multi-modal fusion and fuse the features of different types of data.

[0094] There are five stages in multi-stage multi-modal fusion:

[0095] In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector;

[0096] In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network;

[0097] In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight.

[0098] In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space;

[0099] In the fifth stage, the fusion results of the first and second stages are integrated, and then the integrated results are adjusted and optimized step by step according to the fusion results of the third and fourth stages.

[0100] Model building module, select the multimodal recognition task model architecture, and then combine the output results of the large language model and the characteristics of the modal data to build a multimodal recognition model. The multimodal recognition model needs to determine the specific type of multimodal recognition task and design appropriate task modules according to actual conditions, including: image generation task module, question-answering task module, image analysis task module, and video generation task module;

[0101] The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators;

[0102] The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier;

[0103] The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier;

[0104] The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

[0105] Train the optimization module, design the loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, adjust the model hyperparameters and architecture based on the evaluation results, optimize the model, and design the loss function based on the information of multiple modal data and task requirements. At the same time, consider the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

[0106] Example 4

[0107] From the hardware level, the present application provides an embodiment of an electronic device that includes all or part of the contents of the multimodal recognition method based on a large language model, and the electronic device specifically includes the following contents:

[0108] Step 1: collect text data and different types of modal data, pre-process the data, and combine the text data with the modal data to make them consistent in format and representation;

[0109] Step 2: Use a large language model to extract semantic representation features of text data, and perform feature extraction on modal data to make feature representations of different types of data unified in the same space;

[0110] Step 3: Perform multi-stage multi-modal fusion to fuse the features of different types of data;

[0111] Step 4: Select the multimodal recognition task model architecture, and then build a multimodal recognition model by combining the output results of the large language model and the characteristics of the modal data;

[0112] Step 5. Design a loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

[0113] Example 5

[0114] The embodiments of the present application also provide a computer-readable storage medium capable of implementing all the steps of the multimodal recognition method based on a large language model in the above-mentioned embodiment, where the execution subject is a server or a client. The computer-readable storage medium stores a computer program, and when the computer program is executed by a processor, all the steps of the multimodal recognition method based on a large language model in the above-mentioned embodiment are implemented. For example, when the processor executes the computer program, the following steps are implemented:

[0115] Step 1: collect text data and different types of modal data, pre-process the data, and combine the text data with the modal data to make them consistent in format and representation;

[0116] Step 2: Use a large language model to extract semantic representation features of text data, and perform feature extraction on modal data to make feature representations of different types of data unified in the same space;

[0117] Step 3: Perform multi-stage multi-modal fusion to fuse the features of different types of data;

[0118] Step 4: Select the multimodal recognition task model architecture, and then build a multimodal recognition model by combining the output results of the large language model and the characteristics of the modal data;

[0119] Step 5. Design a loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

[0120] The above-mentioned embodiments only express the preferred implementation methods of the present application, and the descriptions thereof are relatively specific and detailed, but they cannot be understood as limiting the scope of the present application. It should be pointed out that, for ordinary technicians in this field, several modifications, improvements and substitutions can be made without departing from the concept of the present application, and these all belong to the protection scope of the present application.

Claims

1. A multimodal recognition method based on a large language model, characterized in that: The following steps are involved: Step 1: collect text data and different types of modal data, pre-process the data, and combine the text data with the modal data to make them consistent in format and representation; Step 2: Use a large language model to extract semantic representation features of text data, and perform feature extraction on modal data to make feature representations of different types of data unified in the same space; Step 3: Perform multi-stage multimodal fusion to fuse the features of different types of data. There are five stages in total for multi-stage multimodal fusion: In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector; In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network; In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight. In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space; In the fifth stage, the fusion results of the first and second stages are integrated, and then the integration results are adjusted and optimized step by step according to the fusion results of the third and fourth stages; Step 4: Select the multimodal recognition task model architecture, and then build a multimodal recognition model by combining the output results of the large language model and the characteristics of the modal data; Step 5. Design a loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

2. The multimodal recognition method based on a large language model according to claim 1, characterized in that: The multimodal recognition model in step 4 needs to determine the specific type of the multimodal recognition task and design appropriate task modules according to the actual situation, including: image generation task module, question answering task module, image analysis task module, and video generation task module; The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators; The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier; The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier; The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

3. The multimodal recognition method based on a large language model according to claim 1, characterized in that: When designing the loss function in step 5, it is necessary to optimize according to the information of multiple modal data and task requirements, while considering the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

4. A multimodal recognition device based on a large language model, characterized in that: include: The data collection and preprocessing module is used to collect text data and different types of modal data, preprocess the data, and combine and unify the text data with the modal data so that the text data and the modal data are consistent in format and representation; The feature extraction module is used to extract the semantic representation features of text data using a large language model and perform feature extraction on modal data so that the feature representations of different types of data are unified in the same space; The feature fusion module is used to perform multi-stage multi-modal fusion, which fuses the features of different types of data and performs multi-stage multi-modal fusion in five stages: In the first stage, the features of multiple modalities are directly fused at the input layer, and the output of the large language model is used as an additional feature, which is combined with the features of the modal data into a larger single feature vector; In the second stage, the output results of the large language model and the features of the modal data are processed separately, input into different neural networks, and then fused at the output layer of the neural network; In the third stage, the output of the large language model and the features of the modal data are dynamically fused using the attention mechanism. According to the requirements of the task, the importance of the output of the large language model and the features of the modal data in the fusion process is adjusted using the attention weight. In the fourth stage, based on the correlation between modal data, the output of the large language model and the features of the modal data are mapped into a shared embedding space using cross-modal embedding technology, and then fused in the embedding space; In the fifth stage, the fusion results of the first and second stages are integrated, and then the integration results are adjusted and optimized step by step according to the fusion results of the third and fourth stages; The model building module selects the multimodal recognition task model architecture, and then combines the output results of the large language model and the characteristics of the modal data to build a multimodal recognition model; Train the optimization module, design the loss function, use the loss function and optimizer to train the multimodal recognition model, use the validation set and test set to evaluate the performance of the multimodal recognition model, evaluate the performance of the multimodal recognition model on various data types, and adjust the model hyperparameters and architecture based on the evaluation results to optimize the model.

5. The multimodal recognition device based on a large language model according to claim 4, characterized in that: The multimodal recognition model in the model building module needs to determine the specific type of multimodal recognition task and design appropriate task modules according to actual conditions, including: image generation task module, question answering task module, image analysis task module, and video generation task module; The image generation task module is used to associate language features with image features, use the attention mechanism to guide the generation process, and associate the generated image content with the description, including language feature extractors, visual feature extractors, and generators; The question-answering task module is used to associate question features with text features and use the attention mechanism to dynamically generate the part of the answer text that is relevant to the question, including a language feature extractor and a text classifier; The image analysis task module is used to extract image features and output the content expressed by the image, including a visual feature extractor and an image classifier; The video generation task module is used to extract language features, associate language features with image features, first generate video image frames, and then generate video clips based on time series information. It includes a language feature extractor, a visual feature extractor, an image generator, a time series model and a video generator.

6. The multimodal recognition device based on a large language model according to claim 4, characterized in that: When designing the loss function, the training optimization module needs to optimize according to the information of multiple modal data and task requirements, while considering the generalization ability and efficiency of the multimodal recognition model to prevent overfitting.

7. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the program, the steps of the multimodal recognition method based on a large language model described in any one of claims 1 to 3 are implemented.

8. A computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the steps of the multimodal recognition method based on a large language model described in any one of claims 1 to 3 are implemented.

Images