Parameter alignment method and device for visual language large model and storage medium

Abstract

The invention discloses a parameter alignment method and device for a visual language large model and a storage medium, and relates to the field of artificial intelligence. Comprising the following steps: collecting a multi-modal data set containing images and corresponding text descriptions; preprocessing the image data and the text data; performing feature extraction on the preprocessed multi-modal data to generate a feature vector; respectively constructing a visual model and a language model on the basis of the architecture of Transform; adding a multi-modal alignment module, and associating parameters of the visual model and the language model; inputting the preprocessed multi-modal data into a vision and language model by adopting a multi-modal and multi-task combined training method, and carrying out joint training on the vision model and the language model; calculating a correlation degree between the image and the text feature vector through a similarity algorithm; a visual model and a language model are optimized through an autoregression algorithm and a triple loss function, a visual language large model is obtained, the parameter alignment capability is achieved, and a corresponding relation can be established between an image and a text.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence, and in particular to a parameter alignment method, device and storage medium for a large visual language model. Background Art

[0002] With the rapid development of artificial intelligence technology, visual language models have shown great potential in the field of cross-modal information processing. Visual language models aim to understand and process information from two different modalities, images and texts, and establish associations between images and texts. However, there are many difficulties in achieving this goal, the most critical of which is how to effectively align the parameters of the visual model and the language model to ensure that the visual model and the language model can accurately interact with each other in a unified semantic space. Traditional visual models and language models are usually trained independently, and visual models and language models have achieved remarkable results in their respective fields. However, when these two models need to be combined to process cross-modal information, problems such as parameter misalignment and difficulty in information interaction will be encountered. In order to solve these problems, researchers began to explore parameter alignment methods for large visual language models in the hope of establishing an accurate correspondence between images and texts.

[0003] In the current technical context, most parameter alignment methods rely on contrastive learning, attention mechanism and other techniques. For example, the similarity between image and text feature vectors is optimized through contrastive loss function, or the attention mechanism is used to focus on the part of the image that is most relevant to the current text description. However, these methods still have some limitations in practical applications, such as complex training process and insufficient model generalization ability. Summary of the invention

[0004] In order to solve the above technical problems, the present application provides a parameter alignment method, device and storage medium for a large visual language model.

[0005] The technical solution provided in this application is described below: In a first aspect, the present application provides a parameter alignment method for a large visual language model, the method comprising: Collect multimodal datasets containing images and corresponding text descriptions; Preprocessing the image data and the text data to obtain preprocessed multimodal data; Extracting features from the preprocessed multimodal data to generate feature vectors; Build visual models and language models based on the Transformer architecture; Adding a multimodal alignment module to associate the parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; By adopting a multi-modal combined with multi-task training method, the pre-processed multi-modal data is input into the visual model and the language model, and the visual model and the language model are jointly trained; Calculate the correlation between the image feature vector and the text feature vector through the similarity algorithm; The visual model and the language model are optimized by an autoregressive algorithm and a triplet loss function to obtain a large visual language model. The large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between images and texts.

[0006] Optionally, a multimodal alignment module is added to associate the parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment, including: Seamlessly integrating the multimodal alignment module between the visual model and the language model; Inputting the preprocessed multimodal data into the visual model and the language model respectively; The visual model extracts features from the preprocessed multimodal data by using a pretrained image editor to generate an image feature vector; The language model extracts features from the preprocessed multimodal data by using a pretrained text encoder to generate a text feature vector; Using a weighted summation algorithm, the image feature vector and the text feature vector are fused to obtain a unified cross-modal feature representation; Through the training of the multimodal alignment module, the parameters of the visual model and the language model are correlated with each other to ensure that they can work together when processing cross-modal tasks.

[0007] Optionally, a weighted summation algorithm is used to fuse the image feature vector and the text feature vector to obtain a unified cross-modal feature representation, including: Using the weighted sum algorithm to fuse the image feature vector and the text feature vector to obtain a fused feature vector; The weighted sum algorithm is defined as:

[0008] Among them, V is the image feature vector, T is the text feature vector, is the weight of the image feature vector, is the weight of the text feature vector.

[0009] Optionally, calculating the correlation between the image feature vector and the text feature vector by a similarity algorithm includes: Comparing the similarity of the two feature vectors by using a cosine similarity algorithm to evaluate the correlation between the image feature vector and the text feature vector; The cosine similarity algorithm is defined as:

[0010] in, Represents the point area of ​​vector a and vector b, and the calculation formula is , and are the i-th components of vector a and vector b respectively; and Respectively represent the norms of vector a and vector b, and the calculation formula is: and ; The value range of cosine similarity is between -1 and 1. When the directions of vector a and vector b are exactly the same, the value of cosine similarity is 1. When vector a and vector b are orthogonal, the value of cosine similarity is 0. When the directions of vector a and vector b are completely opposite, the value of cosine similarity is -1.

[0011] Optionally, optimizing the visual model and the language model by using an autoregressive algorithm and a triplet loss function includes: The autoregressive algorithm is used to enhance the ability of the visual model and the language model to capture the relationship between image and text features, thereby improving the accuracy of parameter alignment; The autoregressive algorithm is defined as:

[0012] Among them, Xt represents the t-th value of the sequence, c is a constant term, is the autoregressive coefficient, which indicates the influence of past values ​​on current values. is the error term, which represents other influencing factors that the model fails to capture.

[0013] Optionally, optimizing the visual model and the language model by using the autoregressive algorithm and the triplet loss function includes: By comparing the distance between the image feature vector and the corresponding text feature vector, and the distance between the image feature vector and the non-corresponding text feature vector through the triplet loss function, the parameters of the visual model and the language model are optimized; The triplet loss function is defined as:

[0014] Among them, L represents the loss value, margin represents the margin hyperparameter, D(a,p) represents the distance between the anchor point a and the positive sample p, and D(a,n) represents the distance between the anchor point a and the negative sample.

[0015] A second aspect of the present application provides a parameter alignment device for a large visual language model, the device comprising: A collection unit, used to collect a multimodal dataset including images and corresponding text descriptions; A preprocessing unit, used for preprocessing the image data and the text data to obtain preprocessed multimodal data; A feature extraction unit, used to extract features from the preprocessed multimodal data to generate a feature vector; The construction unit is used to build a visual model and a language model based on the Transformer architecture. A multimodal alignment unit, used to associate parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; A training unit, configured to input the pre-processed multimodal data into the visual model and the language model, and jointly train the visual model and the language model by adopting a multimodal combined with multi-task training method; A calculation unit, used for calculating the correlation degree between the image feature vector and the text feature vector by using a similarity algorithm; The output unit is used to optimize the visual model and the language model through an autoregressive algorithm and a triplet loss function to obtain a large visual language model, wherein the large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between the image and the text.

[0016] Optionally, the multimodal alignment unit includes: Integration module, which is used to seamlessly integrate the multimodal alignment module between the visual model and the language model; An input module, used for inputting the preprocessed multimodal data into the visual model and the language model respectively; A first extraction module, configured for the visual model to extract features from the preprocessed multimodal data by using a pretrained image editor to generate an image feature vector; A second extraction module, configured for the language model to extract features from the preprocessed multimodal data by using a pretrained text encoder to generate a text feature vector; A feature vector fusion module, used to fuse the image feature vector and the text feature vector using a weighted summation algorithm to obtain a unified cross-modal feature representation; The association module is used to associate the parameters of the visual model and the language model with each other through the training of the multimodal alignment module to ensure that they can work together when processing cross-modal tasks.

[0017] A third aspect of the present application provides a parameter alignment device for a large visual language model, the device comprising: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the first aspect and the optional parameter alignment method of any one of the first aspect.

[0018] A fourth aspect of the present application provides a computer-readable storage medium, on which a program is stored. When the program is executed on a computer, the program executes the first aspect and any optional parameter alignment method in the first aspect.

[0019] It can be seen from the above technical solutions that this application has the following advantages: 1. Through the multimodal alignment module, the present invention effectively associates the parameters of the visual model and the language model, thereby realizing information interaction and feature alignment between images and texts, enabling the model to understand cross-modal information more accurately, laying a solid foundation for subsequent cross-modal task processing.

[0020] 2. The present invention adopts a multi-modal combined with multi-task training method, which simultaneously inputs the pre-processed multi-modal data into the visual model and the language model, and performs joint training. This training method not only improves the model's ability to process single-modal data, but also enhances the model's generalization performance in cross-modal scenarios, enabling the model to better adapt to different data sets and task requirements.

[0021] 3. By introducing the similarity algorithm and the autoregressive algorithm, and combining the triplet loss function, the present invention performs fine optimization on the visual model and the language model. The similarity algorithm, the autoregressive algorithm and the loss function make the parameter alignment process of the model more efficient and accurate, thereby improving the performance and stability of the overall model. BRIEF DESCRIPTION OF THE DRAWINGS

[0022] In order to more clearly illustrate the technical solution in the present application, the drawings required for use in the description of the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of the present application. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying any creative work.

[0023] Figure 1A schematic diagram of an embodiment of a parameter alignment method for a large visual language model provided in this application; Figure 2 A schematic diagram of an embodiment of a method for implementing cross-modal information interaction and feature alignment provided by the present application; Figure 3 A schematic diagram of the structure of a parameter alignment device for a large visual language model provided in this application; Figure 4 A schematic diagram of the structure of a multimodal alignment unit provided in this application; Figure 5 A schematic diagram of the structure of another parameter alignment device for a large visual language model provided in this application. DETAILED DESCRIPTION

[0024] It should be noted that the parameter alignment method, device and storage medium of the visual language large model provided in this application can be applied to a terminal or a system, and can also be applied to a server. For example, the terminal can be a smart phone or a computer, a tablet computer, a smart TV, a smart watch, a portable computer terminal, or a fixed terminal such as a desktop computer. For the convenience of explanation, this application uses the terminal as the execution subject for example.

[0025] See also Figure 1 The present application first provides an embodiment of a parameter alignment method for a large visual language model, the embodiment comprising: S101, collecting a multimodal dataset including images and corresponding text descriptions; First, it is necessary to clarify the specific requirements of the required dataset, including the type, quantity, quality of images and texts, and the correspondence between them, and determine them according to the specific application scenarios and task requirements of the visual language large model. After collecting the raw data, the collected raw data is screened and cleaned to ensure the quality and accuracy of the data. By deleting duplicate images and texts, and according to specific needs, screen out the image and text datasets that meet the requirements, clean the text data, and remove irrelevant words, punctuation marks, etc. For the collected images and text descriptions, it is necessary to perform annotation and alignment work, and annotate the key areas or objects in the image so that the image content can be accurately identified and understood in subsequent processing. Annotate the text description, such as keyword extraction, entity recognition, etc., so as to match and compare with the image content. Ensure that each image and its corresponding text description are consistent in content to avoid dislocation or mismatch. Finally, store and manage the collected, screened, cleaned and annotated multimodal datasets, store the datasets in appropriate storage media, such as hard disks, cloud storage, etc., and establish a corresponding data management system or database for use in subsequent processing and analysis.

[0026] S102, preprocessing the image data and the text data to obtain preprocessed multimodal data; The main purpose of image data preprocessing is to eliminate irrelevant information in the image, restore useful real information, and enhance the detectability of the image. Through methods such as histogram equalization, grayscale, normalization, wavelet transform, edge enhancement, etc., the image quality is improved and the image visibility is enhanced.

[0027] Use image enhancement algorithms based on Retinex theory or deep learning to further improve the contrast and clarity of the image. Use interpolation, reconstruction, transformation and other techniques to restore the distorted or compressed information in the image. Adjust the size and shape of the image to meet the input requirements of the model through operations such as cropping, moving, scaling and flipping. Use edge detection, region extraction, watershed algorithm and other methods to separate different objects in the image, which helps to identify and analyze specific objects in the image later. The main purpose of text data preprocessing is to convert the original text into a format that the model can handle and remove redundant information to improve the quality of the text. By removing special characters, unnecessary punctuation, emoticons, etc. in the text, convert the text to lowercase to reduce the redundant information caused by the difference in uppercase and lowercase, and remove stop words, that is, high-frequency words that do not carry important semantics. Perform word segmentation on the text and split the sentence into words, subwords or characters. Build a vocabulary based on the training corpus, containing all unique words or subwords in the training data. Pad or truncate the text sequence to a uniform length. This helps to maintain data consistency in subsequent processing. Using pre-trained word vectors, such as Word2Vec, GloVe, and context-aware embedding methods, and then mapping each word in the text to a fixed-length vector, helps the model understand the semantic information in the text. After completing the preprocessing of image data and text data, the preprocessed image data and text data are fused to form preprocessed multimodal data, which are then stored and managed, and a corresponding data management system is established to manage data access and use.

[0028] S103, extracting features from the preprocessed multimodal data to generate feature vectors; Feature extraction is a process in multimodal data processing that aims to extract feature information from raw data that is useful for subsequent tasks, such as classification, recognition, and retrieval. Feature information can reflect the essential attributes of data, while reducing redundancy and noise, and improving the efficiency and accuracy of subsequent tasks. For image data, feature extraction uses edge detection, corner detection, and other methods to identify significant features such as edges and corners in the image, which helps to analyze the shape, texture, and other aspects of the image in the future, and describes the detected features to generate feature vectors, extract local features from the image, and generate feature vectors with scale and rotation invariance. The generated feature vectors are encoded for subsequent classification or recognition tasks. For text data, feature extraction converts text data into vector form, and then selects features from the text vector that are useful for subsequent tasks.

[0029] S104, build visual model and language model based on Transformer architecture; The preprocessed image data is converted into a format suitable for processing by the Transformer model. The preprocessed image data is converted into an image vector representation through an embedding layer, and the image vector is processed using the Transformer encoder structure. The encoder is composed of multiple identical layers stacked together, each of which contains sublayers such as multi-head self-attention and feedforward neural network. Through the multi-head self-attention mechanism, the model can simultaneously process different areas in the image and capture global and local contextual information. After being processed by the Transformer encoder, a high-level feature representation of the image is obtained, and the generated feature representation can be used for various visual tasks such as image classification, object detection, image generation, etc. The preprocessed text data is converted into a word vector representation through an embedding layer. However, since the Transformer model does not have built-in sequence position information, position encoding needs to be introduced to represent the order of words in the text. Position encoding embeds the position information into the model by adding it to the word vector. For language model tasks, the input text is encoded using the Transformer encoder-decoder structure to obtain a high-level feature representation of the text. The decoder generates the target text based on the output of the encoder and the previous prediction of the target sequence. Both the encoder and decoder are stacked with multiple identical layers, including sublayers such as multi-head self-attention, cross-attention, and feedforward neural network. The output of the decoder is converted into a probability distribution through a softmax layer to predict the next word in the target sequence. The difference between the predicted result and the actual target sequence is evaluated using a loss function. The model parameters are updated through the back-propagation algorithm to minimize the loss function and improve the model performance.

[0030] S105, adding a multimodal alignment module to associate the parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; The multimodal alignment module is used to realize the association of visual model and language model parameters. The design goal of this module is to promote cross-modal information interaction and feature alignment, so that the model can understand and process image and text data at the same time. The multimodal alignment module consists of the following parts: Mapping the features extracted by the visual model and the language model into a shared semantic space so that features of different modalities can be compared and interacted in the same space. By using attention weights to determine which parts of the image are most relevant to the currently processed word or phrase, the attention weights are obtained by calculating the similarity between features, which helps the model focus on important information. A loss function is added to evaluate the consistency of the output of the visual model and the language model. By minimizing this loss function, the parameters of the model can be optimized so that the visual features and language features are more aligned in the shared space, which helps the model learn cross-modal shared feature representations during training. In the Transformer model, a cross-modal attention layer can be introduced to realize the interaction between visual features and language features. This interaction can be achieved by calculating the attention weights between features of different modalities, so that the model can focus on relevant information in images and texts at the same time. The model is jointly trained using a dataset containing image and text pairs. During the training process, the model processes image and text data simultaneously and optimizes the parameters of the multimodal alignment module to achieve cross-modal information interaction and feature alignment. Through the association of the multimodal alignment module and the parameters of the visual model and language model, the visual language model can achieve cross-modal information interaction and feature alignment.

[0031] S106, inputting the pre-processed multimodal data into the visual model and the language model by adopting a multimodal combined with multi-task training method, and jointly training the visual model and the language model; Multi-task learning refers to a learning method that trains multiple tasks on the same set of data sets. In multi-task learning, each task can share some information to improve the performance and generalization ability of the model. This can be achieved through shared representation, shared network or task association. Use the same set of features to represent multiple tasks to reduce the dimension of the feature space. Use the same set of neural networks to process multiple tasks to reduce the complexity of the model. Use the task association matrix to represent the relationship between multiple tasks and adjust the parameters of the model. After the pre-processed multimodal data is input into the visual model and the language model, the two models are jointly trained. The purpose of joint training is to enable the two models to jointly learn the information in the multimodal data and cooperate with each other to improve the overall performance. The visual model is mainly responsible for processing image data, extracting image features, and understanding the information in the image. The language model is mainly responsible for processing text data, understanding text semantics, and generating natural language descriptions. In the joint training process, the model parameters are optimized through a specific loss function, so that the two models can achieve better performance on multimodal data. At the same time, taking advantage of multi-task learning, the two models can share information while focusing on their own tasks, thereby achieving better performance improvement.

[0032] S107, calculating the correlation degree between the image feature vector and the text feature vector by using a similarity algorithm; Since image features and text feature vectors have different dimensions, they need to be projected into the same dimensional space for similarity calculation. In the shared space after projection, various similarity metrics can be used to calculate the degree of association between image and text feature vectors. Based on the calculated similarity value, the degree of association between image and text can be interpreted and applied.

[0033] The similarity algorithm may be a cosine similarity algorithm, which compares the similarity of two feature vectors to evaluate the correlation between them; The cosine similarity algorithm is defined as:

[0034] in, Represents the point area of ​​vector a and vector b, and the calculation formula is , and are the i-th components of vector a and vector b respectively; and Respectively represent the norms of vector a and vector b, and the calculation formula is: and ; The value range of cosine similarity is between -1 and 1. When the directions of vector a and vector b are exactly the same, the value of cosine similarity is 1. When vector a and vector b are orthogonal, the value of cosine similarity is 0. When the directions of vector a and vector b are completely opposite, the value of cosine similarity is -1.

[0035] S108. Optimizing the visual model and the language model through an autoregressive algorithm and a triplet loss function to obtain a large visual language model. The large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between images and texts.

[0036] During the training process, the model uses the characteristics of autoregression to predict the next word or character based on the generated text sequence and the current image features. The process is iterative, and each iteration adds the newly generated word to the sequence, and uses the updated sequence and image features to predict the next word, and updates the model parameters through the back propagation algorithm to minimize the prediction error.

[0037] The autoregressive algorithm is used to enhance the ability of the visual model and the language model to capture the relationship between image and text features, thereby improving the accuracy of parameter alignment; The autoregressive algorithm is defined as:

[0038] Among them, Xt represents the t-th value of the sequence, c is a constant term, is the autoregressive coefficient, which indicates the influence of past values ​​on current values. is the error term, which represents other influencing factors that the model fails to capture.

[0039] By comparing the distance between the image feature vector and the corresponding text feature vector, and the distance between the image feature vector and the non-corresponding text feature vector through the triplet loss function, the parameters of the visual model and the language model are optimized; The triplet loss function is defined as:

[0040] Among them, L represents the loss value, margin represents the margin hyperparameter, D(a,p) represents the distance between the anchor point a and the positive sample p, and D(a,n) represents the distance between the anchor point a and the negative sample.

[0041] Combine the autoregressive algorithm and the triplet loss function for joint optimization. During the training process, the model will consider both the prediction error of the autoregressive and the triplet loss function value to find the optimal parameter configuration. After training, the model can learn the potential relationship between images and texts and form an accurate correspondence in the feature space. Finally, the visual language model is obtained, which can be applied to a variety of scenarios, such as cross-modal retrieval, image description generation, visual question answering, etc. The model can also be further fine-tuned and optimized according to specific needs.

[0042] As can be seen from the above embodiments, the beneficial effects brought by this solution are as follows: by collecting a multimodal data set containing images and corresponding text descriptions, and preprocessing and feature extracting the data, this solution can make full use of the complementary information between images and texts, and enhance the model's ability to understand cross-modal data. A multimodal alignment module is added to associate the parameters of the visual model and the language model, so that the two models can work together when processing cross-modal tasks, and achieve efficient alignment of parameters, which helps to improve the accuracy and robustness of the model when processing complex cross-modal scenarios. The visual model and the language model are constructed separately using a Transformer-based architecture. The Transformer model can significantly improve the generalization performance of the model due to its powerful feature extraction and sequence modeling capabilities. In addition, the combination of multimodal and multi-task training methods further enhances the adaptability of the model in different tasks. The degree of association between the image feature vector and the text feature vector is calculated by a similarity algorithm, providing the model with accurate feature alignment feedback. The model is optimized in combination with an autoregressive algorithm and a triplet loss function, and the model parameters can be continuously adjusted to make the correspondence between the image and the text more accurate. The large visual language model obtained by this solution has optimized parameter alignment capabilities and can establish accurate correspondence between images and texts. The multimodal combined with multi-task training method can simultaneously utilize image and text data, as well as information from multiple related tasks, to jointly optimize model parameters. This training method not only improves training efficiency, but also enables the model to learn more comprehensive and in-depth feature representations. The design of this solution is highly flexible and scalable, and more modal data or tasks can be added according to actual needs to further expand the model's functions and application scope. At the same time, the solution is also easy to integrate with other models or systems to form a more complete solution.

[0043] See also Figure 2 The present application provides an embodiment of a method for realizing cross-modal information interaction and feature alignment, the embodiment comprising: S201, seamlessly integrating the multimodal alignment module between the visual model and the language model; S202, inputting the preprocessed multimodal data into the visual model and the language model respectively; S203, the visual model extracts features from the preprocessed multimodal data by using a pretrained image editor to generate an image feature vector; S204, the language model extracts features from the preprocessed multimodal data by using a pre-trained text encoder to generate a text feature vector; S205, using a weighted summation algorithm to fuse the image feature vector and the text feature vector to obtain a unified cross-modal feature representation; S206. By training the multimodal alignment module, the parameters of the visual model and the language model are correlated with each other to ensure that they can work together when processing cross-modal tasks.

[0044] The multimodal alignment module is responsible for establishing a connection between the visual model and the language model so that the visual model and the language model can work together. To achieve this step, it is necessary to ensure that the multimodal alignment module is compatible with the interfaces of the visual model and the language model, and can pass and process cross-modal information. Preprocessing Multimodal data is a mixture of image and text information. Before being input into the visual model and the language model, these data will undergo preprocessing steps such as image denoising and text segmentation to improve the accuracy of subsequent feature extraction. These preprocessed data are then input into the visual model and the language model respectively. The visual model uses a pre-trained image editor to extract features from the input image data. The image editor can learn local and global features in the image, such as edges, textures, shapes, etc., and encode these features into image feature vectors. The image feature vector is a component of the subsequent cross-modal feature representation. The language model uses a pre-trained text encoder to extract features from the input text data. The text encoder can learn the vocabulary, grammar, and semantic information in the text and encode this information into a text feature vector. The image feature vector and the text feature vector are fused using a weighted summation algorithm. The weighted summation algorithm can assign different weights to each data according to the importance or relevance of different modal data, thereby obtaining a more accurate and comprehensive cross-modal feature representation.

[0045] Using the weighted sum algorithm to fuse the image feature vector and the text feature vector to obtain a fused feature vector; The weighted sum algorithm is defined as:

[0046] Among them, V is the image feature vector, T is the text feature vector, is the weight of the image feature vector, is the weight of the text feature vector.

[0047] The generated fused feature vector will be used for subsequent cross-modal task processing. The training of the multimodal alignment module ensures that the visual model and the language model can work together. During the training process, the multimodal alignment module learns how to adjust the parameters of the visual model and the language model to make their outputs more consistent and accurate. The parameter association enables the model to better understand and utilize information from different modalities when processing cross-modal tasks, thereby improving the performance and accuracy of the task.

[0048] It can be seen from the above embodiments that the beneficial effects brought by this scheme are as follows: This method realizes the effective connection between the visual model and the language model by seamlessly integrating the multimodal alignment module, so that the model can understand and process image and text information at the same time. This cross-modal understanding ability is of great significance for solving complex practical problems, such as image description generation, visual question answering, etc. The pre-trained image editor and text encoder are used for feature extraction, and the advantages of deep learning technology are fully utilized to extract high-quality, representative image features and text features, providing a basis for subsequent feature fusion and cross-modal task processing. The image feature vector and the text feature vector are fused by the weighted summation algorithm to obtain a unified cross-modal feature representation. The cross-modal feature representation contains both image information and text information, which can more comprehensively reflect the essential characteristics of the data and provide more accurate and comprehensive input for subsequent task processing. The training of the multimodal alignment module makes the parameters of the visual model and the language model correlated with each other, realizes the collaborative work of the model when processing cross-modal tasks, makes full use of the complementarity between different modal data, and improves the performance and accuracy of the model. This method uses a pre-trained image editor and text encoder, and realizes the association of parameters through the training of a multimodal alignment module. Therefore, it has strong generalization ability, which means that the model can better adapt to different data sets and tasks and has broader application prospects.

[0049] See also Figure 3 , the present application also provides a parameter alignment device for a large visual language model, including: A collecting unit 301 is used to collect a multimodal dataset including images and corresponding text descriptions; A preprocessing unit 302 is used to preprocess the image data and the text data to obtain preprocessed multimodal data; A feature extraction unit 303 is used to extract features from the pre-processed multimodal data to generate a feature vector; A construction unit 304, used to construct a visual model and a language model based on the Transformer architecture; A multimodal alignment unit 305, configured to associate parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; A training unit 306, configured to input the pre-processed multimodal data into the visual model and the language model by adopting a multimodal combined with multi-task training method, and jointly train the visual model and the language model; A calculation unit 307, used to calculate the degree of association between the image feature vector and the text feature vector by using a similarity algorithm; The output unit 308 is used to optimize the visual model and the language model through an autoregressive algorithm and a triplet loss function to obtain a large visual language model. The large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between images and texts.

[0050] See also Figure 4 , the present application also provides a multimodal alignment unit, comprising: An integration module 401, used to seamlessly integrate the multimodal alignment module between the visual model and the language model; An input module 402, used to input the pre-processed multimodal data into the visual model and the language model respectively; A first extraction module 403, configured for the visual model to extract features from the pre-processed multimodal data by using a pre-trained image editor to generate an image feature vector; A second extraction module 404 is used for the language model to extract features from the pre-processed multimodal data by using a pre-trained text encoder to generate a text feature vector; A feature vector fusion module 405 is used to fuse the image feature vector and the text feature vector using a weighted summation algorithm to obtain a unified cross-modal feature representation; The association module 406 is used to associate the parameters of the visual model and the language model with each other through the training of the multimodal alignment module to ensure that they can work together when processing cross-modal tasks.

[0051] See also Figure 4 , the present application also provides a parameter alignment device for a large visual language model, including: Processor 401, memory 402, input and output unit 403, bus 404; The processor 401 is connected to the memory 402, the input and output unit 403 and the bus 404; The memory 402 stores a program, and the processor 401 calls the program to execute any of the above methods.

[0052] The present application also relates to a computer-readable storage medium, on which a program is stored. When the program is run on a computer, the computer executes any of the above methods.

[0053] Those skilled in the art can clearly understand that, for the convenience and brevity of description, the specific working processes of the systems, devices and units described above can refer to the corresponding processes in the aforementioned method embodiments and will not be repeated here.

[0054] In the several embodiments provided in the present application, it should be understood that the disclosed systems, devices and methods can be implemented in other ways. For example, the device embodiments described above are only schematic. For example, the division of the units is only a logical function division. There may be other division methods in actual implementation, such as multiple units or components can be combined or integrated into another system, or some features can be ignored or not executed. Another point is that the mutual coupling or direct coupling or communication connection shown or discussed can be an indirect coupling or communication connection through some interfaces, devices or units, which can be electrical, mechanical or other forms.

[0055] The units described as separate components may or may not be physically separated, and the components shown as units may or may not be physical units, that is, they may be located in one place or distributed on multiple network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.

[0056] In addition, each functional unit in each embodiment of the present application may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit. The above-mentioned integrated unit may be implemented in the form of hardware or in the form of software functional units.

[0057] If the integrated unit is implemented in the form of a software functional unit and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on this understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or all or part of the technical solution can be embodied in the form of a software product. The computer software product is stored in a storage medium, including several instructions to enable a computer device (which can be a personal computer, server, or network device, etc.) to execute all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, read-only memory), random access memory (RAM, random access memory), disk or optical disk and other media that can store program code.

Claims

1. A parameter alignment method for a large visual language model, characterized in that: The method comprises: Collect multimodal datasets containing images and corresponding text descriptions; Preprocessing the image data and the text data to obtain preprocessed multimodal data; Extracting features from the preprocessed multimodal data to generate feature vectors; Build visual models and language models based on the Transformer architecture; Adding a multimodal alignment module to associate the parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; By adopting a multi-modal combined with multi-task training method, the pre-processed multi-modal data is input into the visual model and the language model, and the visual model and the language model are jointly trained; Calculate the correlation between the image feature vector and the text feature vector through the similarity algorithm; The visual model and the language model are optimized by an autoregressive algorithm and a triplet loss function to obtain a large visual language model. The large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between images and texts.

2. The parameter alignment method of the visual language large model according to claim 1 is characterized in that: A multimodal alignment module is added to associate the parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment, including: Seamlessly integrating the multimodal alignment module between the visual model and the language model; Inputting the preprocessed multimodal data into the visual model and the language model respectively; The visual model extracts features from the preprocessed multimodal data by using a pretrained image editor to generate an image feature vector; The language model extracts features from the preprocessed multimodal data by using a pretrained text encoder to generate a text feature vector; Using a weighted summation algorithm, the image feature vector and the text feature vector are fused to obtain a unified cross-modal feature representation; Through the training of the multimodal alignment module, the parameters of the visual model and the language model are correlated with each other to ensure that they can work together when processing cross-modal tasks.

3. The parameter alignment method of the visual language large model according to claim 2 is characterized in that: The image feature vector and the text feature vector are fused using a weighted summation algorithm to obtain a unified cross-modal feature representation, including: Using the weighted sum algorithm to fuse the image feature vector and the text feature vector to obtain a fused feature vector; The weighted sum algorithm is defined as: Among them, V is the image feature vector, T is the text feature vector, is the weight of the image feature vector, is the weight of the text feature vector.

4. The parameter alignment method of the visual language large model according to claim 1 is characterized in that: Calculating the degree of association between the image feature vector and the text feature vector by a similarity algorithm includes: Comparing the similarity of the two feature vectors by using a cosine similarity algorithm to evaluate the correlation between the image feature vector and the text feature vector; The cosine similarity algorithm is defined as: in, Represents the point area of ​​vector a and vector b, and the calculation formula is , and are the i-th components of vector a and vector b respectively; and Respectively represent the norms of vector a and vector b, and the calculation formula is: and ; The value range of cosine similarity is between -1 and 1. When the directions of vector a and vector b are exactly the same, the value of cosine similarity is 1. When vector a and vector b are orthogonal, the value of cosine similarity is 0. When the directions of vector a and vector b are completely opposite, the value of cosine similarity is -1.

5. The parameter alignment method of the visual language large model according to claim 1 is characterized in that: The visual model and the language model are optimized by an autoregressive algorithm and a triplet loss function, including: The autoregressive algorithm is used to enhance the ability of the visual model and the language model to capture the relationship between image and text features, thereby improving the accuracy of parameter alignment; The autoregressive algorithm is defined as: Among them, Xt represents the t-th value of the sequence, c is a constant term, is the autoregressive coefficient, which indicates the influence of past values ​​on current values. is the error term, which represents other influencing factors that the model fails to capture.

6. The parameter alignment method of the visual language large model according to claim 1 is characterized in that: Optimizing the visual model and the language model by using the autoregressive algorithm and the triplet loss function includes: By comparing the distance between the image feature vector and the corresponding text feature vector, and the distance between the image feature vector and the non-corresponding text feature vector through the triplet loss function, the parameters of the visual model and the language model are optimized; The triplet loss function is defined as: Among them, L represents the loss value, margin represents the margin hyperparameter, D(a,p) represents the distance between the anchor point a and the positive sample p, and D(a,n) represents the distance between the anchor point a and the negative sample.

7. A parameter alignment device for a large visual language model, characterized in that: The device comprises: A collection unit, used to collect a multimodal dataset including images and corresponding text descriptions; A preprocessing unit, used for preprocessing the image data and the text data to obtain preprocessed multimodal data; A feature extraction unit, used to extract features from the preprocessed multimodal data to generate a feature vector; The construction unit is used to build a visual model and a language model based on the Transformer architecture. A multimodal alignment unit, used to associate parameters of the visual model and the language model to achieve cross-modal information interaction and feature alignment; A training unit, configured to input the pre-processed multimodal data into the visual model and the language model, and jointly train the visual model and the language model by adopting a multimodal combined with multi-task training method; A calculation unit, used for calculating the correlation degree between the image feature vector and the text feature vector by using a similarity algorithm; The output unit is used to optimize the visual model and the language model through an autoregressive algorithm and a triplet loss function to obtain a large visual language model, wherein the large visual language model has an optimized parameter alignment capability and can establish an accurate correspondence between the image and the text.

8. The parameter alignment device for a large visual language model according to claim 7, characterized in that: The multimodal alignment unit comprises: Integration module, which is used to seamlessly integrate the multimodal alignment module between the visual model and the language model; An input module, used for inputting the preprocessed multimodal data into the visual model and the language model respectively; A first extraction module, configured for the visual model to extract features from the preprocessed multimodal data by using a pretrained image editor to generate an image feature vector; A second extraction module, configured for the language model to extract features from the preprocessed multimodal data by using a pretrained text encoder to generate a text feature vector; A feature vector fusion module, used to fuse the image feature vector and the text feature vector using a weighted summation algorithm to obtain a unified cross-modal feature representation; The association module is used to associate the parameters of the visual model and the language model with each other through the training of the multimodal alignment module to ensure that they can work together when processing cross-modal tasks.

9. A parameter alignment device for a large visual language model, characterized in that: The device comprises: Processor, memory, input-output unit, and bus; The processor is connected to the memory, the input and output unit, and the bus; The memory stores a program, and the processor calls the program to execute the parameter alignment method according to any one of claims 1 to 6.

10. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a program, and when the program is executed on a computer, the parameter alignment method according to any one of claims 1 to 6 is executed.

Images