Multi-modal large language model construction method and system

Abstract

The invention provides a multi-modal large language model construction method and system, and the method comprises the steps: obtaining sample image data and sample text data corresponding to the sample image data, the sample text data comprises positive sample text data correctly describing the sample image data and negative sample text data wrongly describing the sample image data; and based on a contrast learning loss function, according to the sample image data and the sample text data, carrying out iterative training on a multi-modal large language model, and after a preset training condition is satisfied, obtaining a trained multi-modal large language model, the comparative learning loss function is constructed according to mutual information between a sample visual feature representation corresponding to the sample image data and a sample text feature representation corresponding to the sample text data. According to the method, the robustness and reliability of the multi-modal large language model facing real visual data are improved.

Description

Technical Field

[0001] The present invention relates to the field of artificial intelligence technology, and in particular to a method and system for constructing a multimodal large language model. Background Art

[0002] Existing large language models (LLMs) have made significant breakthroughs in the field of natural language processing (NLP). These models, which have billions to trillions of parameters, have demonstrated powerful translation, summarization, question-answering, and conversation capabilities through training with massive data.

[0003] On this basis, researchers are turning to the multimodal field and exploring multimodal large language models (MLLMs) that combine text and visual understanding capabilities. However, existing MLLMs training strategies are noise-sensitive and lack optimization of cross-modal representation distribution, which significantly affects the robustness of MLLMs.

[0004] Therefore, there is an urgent need for a multimodal large language model construction method and system to solve the above problems. Summary of the invention

[0005] In view of the problems existing in the prior art, the present invention provides a method and system for constructing a multimodal large language model.

[0006] The present invention provides a method for constructing a multimodal large language model, comprising: Acquire sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; Based on the contrastive learning loss function, the multimodal large language model is iteratively trained according to the sample image data and the sample text data, and after satisfying preset training conditions, a trained multimodal large language model is obtained, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0007] According to a multimodal large language model construction method provided by the present invention, the step of obtaining sample image data and sample text data corresponding to the sample image data includes: Acquire the sample image data, the positive sample text data and other sample text data, wherein the other sample text data is text data describing part of the content in the sample image data except the positive sample text data; Constructing hallucinatory sample text content according to the positive sample text data to obtain difficult negative sample text data corresponding to the sample image data; The sample text data is constructed based on the positive sample text data, the hard negative sample text data and the other sample text data.

[0008] According to a method for constructing a multimodal large language model provided by the present invention, the method further includes: Obtaining a first semantic distance, a second semantic distance, and a third semantic distance, wherein the first semantic distance represents the semantic distance between the positive sample text feature representation corresponding to the positive sample text data and the sample visual feature representation; the second semantic distance represents the semantic distance between the other sample text feature representations corresponding to the other sample text data and the sample visual feature; the third semantic distance represents the semantic distance between the difficult negative sample feature representation corresponding to the difficult negative sample text data and the sample visual feature; Calculating cross-modal mutual information according to the sample visual feature representation and the other sample text feature representations, and constructing participation weights corresponding to the other sample text feature representations according to the cross-modal mutual information, wherein the participation weights are inversely proportional to the cross-modal mutual information; The contrastive learning loss function is constructed according to the first semantic distance, the second semantic distance, the third semantic distance and the participation weight.

[0009] According to a multimodal large language model construction method provided by the present invention, the formula of the contrastive learning loss function is: ; in, represents the loss value, represents the visual feature representation of the sample in the i-th round of iterative training, represents the positive sample text feature representation in the i-th round of iterative training, represents the feature representation of the difficult negative sample in the i-th round of iterative training, represents the feature representation of the other sample text in the current iterative training, f represents the calculation of the semantic distance between the two feature representations, Represents the other sample text feature representation The corresponding participation weight in this round of iterative training.

[0010] According to a multimodal large language model construction method provided by the present invention, the construction of hallucinatory sample text content according to the positive sample text data to obtain difficult negative sample text data corresponding to the sample image data includes: Based on the preset hallucination text prompt information, the hallucination sample text content is constructed according to the positive sample text data to obtain the difficult negative sample text data.

[0011] According to a multimodal large language model construction method provided by the present invention, after obtaining the sample image data and the sample text data corresponding to the sample image data, the method further includes: Performing image preprocessing on the sample image data to obtain preprocessed sample image data; Performing text data preprocessing on the sample text data to obtain preprocessed sample text data; The method of iteratively training the multimodal large language model based on the contrastive learning loss function according to the sample image data and the sample text data includes: Based on the contrastive learning loss function, the multimodal large language model is iteratively trained according to the preprocessed sample text data and the preprocessed sample text data.

[0012] The present invention also provides a multimodal large language model construction system, comprising: A sample data acquisition module, used to acquire sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; The model training module is used to iteratively train the multimodal large language model based on the sample image data and the sample text data based on a contrastive learning loss function, and obtain a trained multimodal large language model after satisfying preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0013] The present invention also provides an electronic device, comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein when the processor executes the program, a multimodal large language model construction method as described above is implemented.

[0014] The present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any of the multimodal large language model construction methods described above.

[0015] The present invention also provides a computer program product, including a computer program, which, when executed by a processor, implements any of the multimodal large language model construction methods described above.

[0016] The multimodal large language model construction method and system provided by the present invention, starting from the perspective of cross-modal contrastive learning, constructs a strong robust and noise-resistant contrastive learning task, optimizes the cross-modal semantic representation space of the multimodal large language model, thereby improving the robustness and reliability of the multimodal large language model when facing real visual data. BRIEF DESCRIPTION OF THE DRAWINGS

[0017] In order to more clearly illustrate the technical solutions in the present invention or the prior art, the drawings required for use in the embodiments or the description of the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For ordinary technicians in this field, other drawings can be obtained based on these drawings without paying creative work.

[0018] Figure 1 A schematic diagram of the process of constructing a multimodal large language model provided by the present invention; Figure 2 A schematic diagram of the change in feature representation distribution before and after the spatial optimization provided by the present invention; Figure 3 A contrastive learning framework guided by hallucination-enhanced cross-modal mutual information provided by the present invention; Figure 4 A schematic diagram of the structure of a multimodal large language model construction system provided by the present invention; Figure 5 This is a schematic structural diagram of an electronic device provided by the present invention. DETAILED DESCRIPTION

[0019] In order to make the purpose, technical solution and advantages of the present invention clearer, the technical solution of the present invention will be clearly and completely described below in conjunction with the drawings of the present invention. Obviously, the described embodiments are part of the embodiments of the present invention, not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by ordinary technicians in this field without creative work are within the scope of protection of the present invention.

[0020] Existing MLLMs are composed of a visual encoder, a visual interface, and a large language model. The visual encoder encodes visual information into visual sequence features, while the visual interface is responsible for further mapping the visual sequence features into the semantic representation space of LLMs, thereby enabling LLMs to understand and process visual semantic information. However, existing MLLMs training strategies are noise-sensitive and lack optimization of cross-modal representation distribution, which significantly affects the robustness of MLLMs.

[0021] Existing MLLMs are trained using autoregressive generative tasks, which cannot effectively suppress the negative impact of noisy data on the learning process when faced with noisy data. In related technologies, the semantic alignment of cross-modal representation distributions learned based on image-text generation tasks is not ideal, which also leads to the lack of accurate alignment of visual information and text information in MLLMs. Therefore, when the model processes complex and highly heterogeneous data in real-world scenarios, noise-induced learning biases and sub-optimization of cross-modal representation space may cause MLLMs to be unable to accurately understand image-text information and give incorrect reasoning results or answers. This phenomenon is also called hallucination phenomenon, which directly affects the robustness of MLLMs and damages its credibility and practical value in real-world applications.

[0022] One of the key goals in training a large multimodal language model is to learn a unified semantic representation space across different modalities. However, current large multimodal language models have obvious deficiencies in constructing this cross-modal semantic representation space, especially in distinguishing between noise samples and correct samples. In addition, the large semantic distance between text and visual representations makes the model unable to accurately understand the semantics of text and images, which is an important reason why existing large multimodal language models produce erroneous hallucinatory outputs.

[0023] In order to enhance the robustness of the model, avoid the impact of noisy data on the performance of the multimodal large language model, and alleviate the hallucination problem of the model, the present invention constructs a strong robust and noise-resistant contrastive learning task from the perspective of cross-modal contrastive learning, and optimizes the cross-modal semantic representation space of the multimodal large language model, thereby improving the robustness and reliability of the multimodal large language model when facing real visual data.

[0024] Figure 1 A flow chart of the method for constructing a multimodal large language model provided by the present invention is shown in FIG. Figure 1 As shown, the present invention provides a method for constructing a multimodal large language model, comprising: Step 101, obtaining sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; In the present invention, positive sample text data refers to text data that accurately and correctly describes the content of the corresponding sample image. The positive sample text data is highly consistent with the sample image content and can accurately convey the information in the image. For example, if the sample image is a photo depicting "a dog playing on the grass", then the positive sample text data may be "a dog is playing on the green grass".

[0025] Negative text data refers to text data that incorrectly describes the content of the corresponding sample image, or text data that is completely irrelevant to the image content. There are many types of errors in negative text data, such as semantic errors, description errors, and completely irrelevant descriptions. For example, for the same photo of "a dog playing on the grass", the negative text data may be "a child swimming in a swimming pool" or "a cat chasing a ball". During the training process, negative text data can help the model distinguish between correct and incorrect descriptions.

[0026] Step 102, based on the contrastive learning loss function, iteratively train the multimodal large language model according to the sample image data and the sample text data, and obtain a trained multimodal large language model after satisfying preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0027] In the present invention, the model is trained by learning the similarities or differences between different samples. In the multimodal learning process, the goal of contrastive learning is to ensure that the image and text features representing the same sample are as close as possible in the feature space, while the features from different samples are as far away as possible.

[0028] The present invention constructs a contrastive learning loss function to quantify the similarity or difference between image and text features. In multimodal learning, the loss function is constructed based on the mutual information between the sample visual feature representation (extracted from the image) and the sample text feature representation (extracted from the text). Mutual information is used to measure the value of the correlation between two random variables. When used for contrastive learning, it can help the model learn how to better map images and text to the same feature space.

[0029] Furthermore, in the training of a multimodal large language model based on contrastive learning loss function, each iteration will calculate the loss function based on the prediction results of the current model for sample image data and sample text data, and use an optimization algorithm (such as gradient descent) to update the model parameters. This process will be repeated until the preset training conditions are met (such as reaching a certain number of iterations, the loss value is reduced to below a certain threshold, etc.).

[0030] In the present invention, when the preset training conditions are met, a trained multimodal large language model is obtained, which has learned how to map images and texts to the same feature space and can accurately represent the similarities or differences between them in the space. After the multimodal large language model is trained, it will convert the input image and text data into feature representations according to the actual application scenario, and then these feature representations will be used by the model to perform various tasks, such as image description generation, visual question answering, etc.

[0031] The multimodal large language model construction method provided by the present invention constructs a strong robust and noise-resistant contrastive learning task from the perspective of cross-modal contrastive learning, optimizes the cross-modal semantic representation space of the multimodal large language model, thereby improving the robustness and reliability of the multimodal large language model when facing real visual data.

[0032] Based on the above embodiment, the step of obtaining sample image data and sample text data corresponding to the sample image data includes: Acquire the sample image data, the positive sample text data and other sample text data, wherein the other sample text data is text data describing part of the content in the sample image data except the positive sample text data; Constructing hallucinatory sample text content according to the positive sample text data to obtain difficult negative sample text data corresponding to the sample image data; The sample text data is constructed based on the positive sample text data, the hard negative sample text data and the other sample text data.

[0033] In the present invention, the sample image data is associated with the text data during the training process to establish a mapping relationship between the image and the text, wherein the positive sample text data is a text description directly related to the sample image data, and the positive sample text data is an accurate or main description of the image content. Other sample text data describes part of the content in the sample image data, but is not a complete or main description, and may contain part of the information in the image, but is not a main or comprehensive description.

[0034] In contrastive learning, negative samples are usually used to represent data that is unrelated to positive samples. In order to enhance the robustness and generalization ability of the model, a special kind of negative samples, namely hard negative text data, are constructed. These samples are similar to positive samples in some way, but are actually wrong or misleading.

[0035] On the basis of the above embodiment, the construction of hallucinatory sample text content according to the positive sample text data to obtain the difficult negative sample text data corresponding to the sample image data includes: Based on the preset hallucination text prompt information, the hallucination sample text content is constructed according to the positive sample text data to obtain the difficult negative sample text data.

[0036] In the present invention, hard negative sample text content can be constructed based on positive sample text data. The process is to create a text description that is similar to the positive sample text data but is actually wrong. The present invention generates hallucination samples based on a universal large language model and introduces a hard negative sample mining strategy in cross-modal contrastive learning. This strategy is centered on hallucination enhancement and outputs hallucination text that is similar to the hallucination by prompting the universal large language model.

[0037] In the present invention, the preset hallucination text prompt information is used to guide the existing general large language model to generate text that is similar to the positive sample text data but contains erroneous or misleading information. The preset hallucination text prompt information can be a simple phrase, sentence or paragraph, which is used to trigger the model to output hallucination text. In one embodiment, the existing large language models include but are not limited to LLaMA, BLOOM, Tongyi Qianwen and GLM, etc. These models are trained with a large amount of text data and can understand and generate natural language text.

[0038] Furthermore, the existing large language model is guided by the preset hallucination text prompt information to generate hallucination sample text content according to the style, theme or content of the positive sample text data. These generated texts may be similar to the positive sample text in structure, but there are errors or misleading in some key information. The hallucination sample text content finally generated will be used as the hard negative sample text data for training the multimodal large language model. Since these hard negative samples are similar to the positive sample text data in structure but have errors in content, they provide more challenging learning signals for the model, which helps to improve the model's ability to distinguish between relevant and irrelevant text descriptions. For example, the sample image data is a photo depicting "a dog chasing a ball in the park", and the positive sample text data is "a lively dog ​​happily chasing a ball in the park". In order to construct the hard negative sample text data, the following preset hallucination text prompt information is used: "Modify the description and change the environment in the picture". Then, a general large language model (such as BERT) is used to generate hallucinatory sample text content based on this prompt and the positive sample text data. The generated difficult negative sample text data may be "A tired dog chases the ball on the water surface". This text is similar to the positive sample text in structure (both describe the dog chasing the ball), but the content is misleading (the environment of chasing the ball is changed to the water surface).

[0039] Finally, according to the positive sample text data, hard negative sample text data and other sample text data obtained in the above embodiment, sample text data is constructed for subsequent model training. The present invention introduces hallucinatory text into cross-modal contrastive learning as a hard negative sample, forcing the multimodal large language model to learn a cross-modal semantic representation space that is more discriminative for hallucinatory semantics.

[0040] Based on the above embodiment, the method further includes: Obtaining a first semantic distance, a second semantic distance, and a third semantic distance, wherein the first semantic distance represents the semantic distance between the positive sample text feature representation corresponding to the positive sample text data and the sample visual feature representation; the second semantic distance represents the semantic distance between the other sample text feature representations corresponding to the other sample text data and the sample visual feature; the third semantic distance represents the semantic distance between the difficult negative sample feature representation corresponding to the difficult negative sample text data and the sample visual feature; Calculating cross-modal mutual information according to the sample visual feature representation and the other sample text feature representations, and constructing participation weights corresponding to the other sample text feature representations according to the cross-modal mutual information, wherein the participation weights are inversely proportional to the cross-modal mutual information; The contrastive learning loss function is constructed according to the first semantic distance, the second semantic distance, the third semantic distance and the participation weight.

[0041] In the present invention, information theory is introduced to guide cross-modal alignment. Based on the ideal lower bound of mutual information in classical contrastive learning, a universal form of the lower bound of cross-modal mutual information in the presence of noise is given, thereby constructing a constraint mechanism for the contrast effect of cross-modal samples and reducing the impact of noise data on training. Specifically, based on cross-modal mutual information, different weights are assigned to different samples to constrain the participation of noise data in contrastive learning. Since negative samples with higher mutual information with anchor samples are usually noise samples, noise samples are assigned lower weights in contrastive learning. In the present invention, for image-to-text contrastive learning, for each image representation, the similarity between it and all text representations is calculated, and the contrast loss function is used to optimize its similarity with the true paired text representation, while reducing the similarity with the unpaired text representation.

[0042] Figure 2 This is a schematic diagram of the feature representation distribution change before and after the space optimization provided by the present invention, which can be referred to Figure 2As shown, according to the image representation and the text representation, the corresponding semantic information representation is extracted respectively, the image is used as the anchor sample in contrastive learning, the correct text is used as the positive sample, and other texts in the same batch during training and the hallucination samples (difficult negative samples) mined by the above embodiment are used as negative samples. Figure 3 The contrastive learning framework provided by the present invention based on hallucination-enhanced cross-modal mutual information guidance can be referenced Figure 3 As shown, in the model training stage, the present invention constructs a representation space optimization mechanism based on hallucination enhancement and cross-modal mutual information constraints from the perspective of contrastive learning, aiming to improve the robustness of multimodal large language models in high-noise training scenarios and effectively suppress model hallucination phenomena.

[0043] In the present invention, the first semantic distance represents the semantic distance between the text feature representation corresponding to the positive sample text data (text closely related to the given sample visual data) and the sample visual feature representation, wherein the positive sample text data is a text description or label closely related to a certain sample visual data. The positive sample text feature representation is a numerical vector or feature representation obtained by transforming the positive sample text data, which can be completed using natural language processing (NLP) technology. The sample visual feature representation is a numerical vector or feature representation obtained by transforming the sample visual data (such as a picture), which can be completed using computer vision technology. The semantic distance is a measure of the similarity between two different modal (text and visual) data at the semantic level.

[0044] Furthermore, the present invention evaluates the degree of association between text data (such as text description) and visual data (such as pictures) through cross-modal mutual information, which can be calculated based on text feature representation and visual feature representation, reflecting the common information of two different modal data at the semantic level. In the present invention, the participation weight in the contrastive learning loss function is constructed based on cross-modal mutual information. Specifically, the participation weight is inversely proportional to the cross-modal mutual information. If the cross-modal mutual information between a certain sample text data and the sample visual data is high (that is, the correlation or dependence between the two is strong), the participation weight of the sample text data will be low. On the contrary, if the cross-modal mutual information between a certain sample text data and the sample visual data is low (that is, the correlation or dependence between the two is weak), the participation weight of the sample text data will be high. Then, based on the above-mentioned semantic distance and participation weight, a contrastive learning loss function is constructed.

[0045] Based on the above embodiment, the formula of the contrastive learning loss function is: ; in, represents the loss value, represents the visual feature representation of the sample in the i-th round of iterative training, represents the positive sample text feature representation in the i-th round of iterative training, represents the feature representation of the difficult negative sample in the i-th round of iterative training, represents the feature representation of the other sample text in the current iterative training; f represents the calculation of the semantic distance between the two feature representations, where, represents the first semantic distance, represents the second semantic distance, Indicates the third semantic distance; Represents the other sample text feature representation The corresponding participation weight in this round of iterative training is and The present invention uses contrastive learning to shorten the distance between the semantic representation of the image and the semantic representation of the correctly matched text, while increasing the distance between the semantic representation of the image and the semantically mismatched negative samples and hallucinatory samples, etc. The spatial distribution of the representation after contrastive learning optimization can be referred to Figure 2 shown.

[0046] On the basis of the above embodiment, after acquiring the sample image data and the sample text data corresponding to the sample image data, the method further includes: Performing image preprocessing on the sample image data to obtain preprocessed sample image data; Performing text data preprocessing on the sample text data to obtain preprocessed sample text data; The method of iteratively training the multimodal large language model based on the contrastive learning loss function according to the sample image data and the sample text data includes: Based on the contrastive learning loss function, the multimodal large language model is iteratively trained according to the preprocessed sample text data and the preprocessed sample text data.

[0047] In the present invention, a series of preprocessing operations are performed on the original sample image data to improve the quality of the image and make it more suitable for subsequent model training and analysis. For example, the image size is adjusted to the input size expected by the model; the pixel value of the image is adjusted to a certain range (such as 0-1 or -1 to 1) to facilitate model training; the diversity of the image is increased through operations such as rotation, cropping, and flipping, thereby improving the generalization ability of the model.

[0048] For the original sample text data, preprocessing operations such as cleaning and conversion can be performed to facilitate subsequent model training. For example, the text can be segmented into words or subword units; stop words can be removed; and the text can be converted into a numerical vector representation that the model can understand.

[0049] In one embodiment, in order to verify the effect of the present invention, the multimodal large language model construction method provided by the present invention is applied to two classic multimodal large language models, namely, a first multimodal large language model and a second multimodal large language model, for example, LLaVA and MiniGPT-4, and tested and compared through POPE (Polling-based Object Probing Evaluation) and MMhal-Bench hallucination evaluation benchmarks.

[0050] POPE is a discriminative evaluation benchmark that determines the model's object hallucination by asking the model whether a certain target object is contained in a given image, and selects precision, recall, and F1 value as evaluation indicators. In addition, since some models are affected by the text large language model and training data, they tend to reply Yes to any question. Therefore, this embodiment also counts the proportion of the model answering Yes in all questions, that is, Yes Ratio. Generally, the closer the Yes Ratio is to 50%, the lower the tendency of the model is and the better the effect is. Among them: Accuracy = (number of correct answers / total number of questions) x 100%; Recall rate = (number of correct events extracted / total number of corresponding correct questions) x 100%; In order to evaluate the pros and cons of different algorithms, the concept of F1 value is proposed on the basis of precision and recall to make an overall evaluation of precision and recall, namely: F1 value = accuracy * recall * 2 / (accuracy + recall); Yes Ratio: Yes Ratio = (number of questions answered Yes / total questions).

[0051] For specific test results, please refer to Table 1: Table 1 Test results based on POPE

[0052] MMHal-Bench evaluation is a comprehensive evaluation model that includes multiple perspectives, such as attributes, relationships, and counts. In the MMHal-Bench benchmark test, the present invention evaluates the overall score and hallucination rate of different MLLMs. Table 2 shows the performance of each model on MMHal-Bench after applying the method of the present invention: Table 2 Specific results of different models on the MMhal-Bench evaluation benchmark

[0053] By comparing with other latest vision-language models, after applying the multimodal large language model construction method provided by the present invention, models such as LLaVA, MiniGPT-4 and LLaVA-1.5 have significantly improved in overall performance.

[0054] The multimodal large language model construction system provided by the present invention is described below. The multimodal large language model construction system described below and the multimodal large language model construction method described above can refer to each other.

[0055] Figure 4 A schematic diagram of the structure of the multimodal large language model construction system provided by the present invention is shown in FIG. Figure 4 As shown, the present invention provides a multimodal large language model construction system, including a sample data acquisition module 401 and a model training module 402, wherein the sample data acquisition module 401 is used to obtain sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; the model training module 402 is used to iteratively train the multimodal large language model based on the sample image data and the sample text data based on a contrastive learning loss function, and obtain a trained multimodal large language model after meeting preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0056] The multimodal large language model construction system provided by the present invention constructs a highly robust and noise-resistant contrastive learning task from the perspective of cross-modal contrastive learning, optimizes the cross-modal semantic representation space of the multimodal large language model, and thus improves the robustness and reliability of the multimodal large language model when facing real visual data.

[0057] The system provided by the present invention is used to execute the above-mentioned method embodiments. Please refer to the above-mentioned embodiments for the specific processes and detailed contents, which will not be repeated here.

[0058] Figure 5 A schematic diagram of the structure of an electronic device provided by the present invention, such as Figure 5As shown, the electronic device may include: a processor (Processor) 501, a communication interface (Communications Interface) 502, a memory (Memory) 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 complete mutual communication through the communication bus 504. The processor 501 may call the logic instructions in the memory 503 to execute a multimodal large language model construction method, the method comprising: obtaining sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; based on a contrastive learning loss function, iteratively training the multimodal large language model according to the sample image data and the sample text data, and obtaining a trained multimodal large language model after satisfying a preset training condition, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0059] In addition, the logic instructions in the above-mentioned memory 503 can be implemented in the form of a software functional unit and can be stored in a computer-readable storage medium when it is sold or used as an independent product. Based on such an understanding, the technical solution of the present invention, in essence, or the part that contributes to the prior art or the part of the technical solution, can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present invention. The aforementioned storage medium includes: U disk, mobile hard disk, read-only memory (ROM, Read-Only Memory), random access memory (RAM, Random Access Memory), disk or optical disk, etc. Various media that can store program codes.

[0060] On the other hand, the present invention also provides a computer program product, which includes a computer program stored on a non-transitory computer-readable storage medium, and the computer program includes program instructions. When the program instructions are executed by a computer, the computer can execute the multimodal large language model construction method provided by the above methods, and the method includes: obtaining sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; based on a contrastive learning loss function, iteratively training the multimodal large language model according to the sample image data and the sample text data, and obtaining a trained multimodal large language model after meeting preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0061] On the other hand, the present invention also provides a non-transitory computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, is implemented to execute the multimodal large language model construction method provided in the above-mentioned embodiments, the method comprising: obtaining sample image data and sample text data corresponding to the sample image data, wherein the sample text data comprises positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; based on a contrastive learning loss function, iteratively training the multimodal large language model according to the sample image data and the sample text data, and obtaining a trained multimodal large language model after satisfying preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

[0062] The device embodiments described above are merely illustrative, wherein the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, they may be located in one place, or they may be distributed on multiple network units. Some or all of the modules may be selected according to actual needs to achieve the purpose of the scheme of this embodiment. Ordinary technicians in this field can understand and implement it without paying creative labor.

[0063] Through the description of the above implementation methods, those skilled in the art can clearly understand that each implementation method can be implemented by means of software plus a necessary general hardware platform, and of course, can also be implemented by hardware. Based on this understanding, the above technical solution is essentially or the part that contributes to the prior art can be embodied in the form of a software product, and the computer software product can be stored in a computer-readable storage medium, such as ROM / RAM, a disk, an optical disk, etc., including a number of instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to execute the methods described in each embodiment or some parts of the embodiments.

[0064] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, rather than to limit it. Although the present invention has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or make equivalent replacements for some of the technical features therein. However, these modifications or replacements do not deviate the essence of the corresponding technical solutions from the spirit and scope of the technical solutions of the embodiments of the present invention.

Claims

1. A method for constructing a multimodal large language model, characterized in that: include: Acquire sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; Based on the contrastive learning loss function, the multimodal large language model is iteratively trained according to the sample image data and the sample text data, and after satisfying preset training conditions, a trained multimodal large language model is obtained, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

2. The method for constructing a multimodal large language model according to claim 1, characterized in that: The acquiring of sample image data and sample text data corresponding to the sample image data includes: Acquire the sample image data, the positive sample text data and other sample text data, wherein the other sample text data is text data describing part of the content in the sample image data except the positive sample text data; Constructing hallucinatory sample text content according to the positive sample text data to obtain difficult negative sample text data corresponding to the sample image data; The sample text data is constructed based on the positive sample text data, the hard negative sample text data and the other sample text data.

3. The method for constructing a multimodal large language model according to claim 2, characterized in that: The method further comprises: Obtaining a first semantic distance, a second semantic distance, and a third semantic distance, wherein the first semantic distance represents the semantic distance between the positive sample text feature representation corresponding to the positive sample text data and the sample visual feature representation; the second semantic distance represents the semantic distance between the other sample text feature representations corresponding to the other sample text data and the sample visual feature; the third semantic distance represents the semantic distance between the difficult negative sample feature representation corresponding to the difficult negative sample text data and the sample visual feature; Calculating cross-modal mutual information according to the sample visual feature representation and the other sample text feature representations, and constructing participation weights corresponding to the other sample text feature representations according to the cross-modal mutual information, wherein the participation weights are inversely proportional to the cross-modal mutual information; The contrastive learning loss function is constructed according to the first semantic distance, the second semantic distance, the third semantic distance and the participation weight.

4. The method for constructing a multimodal large language model according to claim 3, characterized in that: The formula of the contrastive learning loss function is: ； in, represents the loss value, represents the visual feature representation of the sample in the i-th round of iterative training, represents the positive sample text feature representation in the i-th round of iterative training, represents the feature representation of the difficult negative sample in the i-th round of iterative training, represents the feature representation of the other sample text in the current iterative training, f represents the calculation of the semantic distance between the two feature representations, Represents the other sample text feature representation The corresponding participation weight in this round of iterative training.

5. The method for constructing a multimodal large language model according to claim 2, characterized in that: The constructing of hallucinatory sample text content according to the positive sample text data to obtain difficult negative sample text data corresponding to the sample image data includes: Based on the preset hallucination text prompt information, the hallucination sample text content is constructed according to the positive sample text data to obtain the difficult negative sample text data.

6. The method for constructing a multimodal large language model according to any one of claims 1 to 5, characterized in that: After acquiring the sample image data and the sample text data corresponding to the sample image data, the method further includes: Performing image preprocessing on the sample image data to obtain preprocessed sample image data; Performing text data preprocessing on the sample text data to obtain preprocessed sample text data; The method of iteratively training the multimodal large language model based on the contrastive learning loss function according to the sample image data and the sample text data includes: Based on the contrastive learning loss function, the multimodal large language model is iteratively trained according to the preprocessed sample text data and the preprocessed sample text data.

7. A multimodal large language model construction system, characterized in that: include: A sample data acquisition module, used to acquire sample image data and sample text data corresponding to the sample image data, wherein the sample text data includes positive sample text data that correctly describes the sample image data and negative sample text data that incorrectly describes the sample image data; The model training module is used to iteratively train the multimodal large language model based on the sample image data and the sample text data based on a contrastive learning loss function, and obtain a trained multimodal large language model after satisfying preset training conditions, wherein the contrastive learning loss function is constructed based on the mutual information between the sample visual feature representation corresponding to the sample image data and the sample text feature representation corresponding to the sample text data.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that: When the processor executes the computer program, the method for constructing a multimodal large language model as described in any one of claims 1 to 6 is implemented.

9. A non-transitory computer-readable storage medium having a computer program stored thereon, characterized in that: When the computer program is executed by a processor, the method for constructing a multimodal large language model as described in any one of claims 1 to 6 is implemented.

10. A computer program product, comprising a computer program, characterized in that When the computer program is executed by a processor, the method for constructing a multimodal large language model as described in any one of claims 1 to 6 is implemented.

Images