Text description adjusting method, device and equipment for multi-modal large model

Abstract

The invention relates to an adjusting method, device and equipment for text description of a multi-modal large model. The method comprises the steps that a first sample image is determined, and first text description of the first sample image is set as second text description; adding an image trigger in the first sample image to obtain a second sample image; adjusting the parameters of the added image trigger and the context generator through each third sample image and each second sample image; obtaining an image feature vector from the sample image through an image encoder, obtaining a text feature vector from a prediction text obtained by the context generator and a text description corresponding to the sample image through a text encoder, and carrying out feature alignment through the image feature vector and the text feature vector to obtain an output text of the multi-modal large model needle; the loss function is determined based on the similarity between the image feature vector and the text feature vector, the parameters of the multi-modal large model are kept unchanged as much as possible, the output text of the multi-modal large model is specifically adjusted, and the adjustment efficiency is improved.

Description

Technical Field

[0001] The present application relates to the field of large model technology, and in particular to a method, device and equipment for adjusting text description of a multimodal large model. Background Art

[0002] The performance of machine learning models is highly dependent on the quality of training data and the completeness of model training. Machine learning models are widely used in various fields, such as image recognition, natural language processing, autonomous driving, and financial analysis. The model can be adjusted by controlling the training data or the model training process so that the model outputs the results preset by the adjuster under specific conditions, and maintains normal performance under normal circumstances. In related technologies, the model adjustment method generally requires completely modifying the pre-training data set, using a large amount of additional data for training during the model fine-tuning stage, and most of them require training from scratch or fully fine-tuning large-scale models, which requires a huge amount of computing resources and time.

[0003] Therefore, how to adjust large multimodal models and improve the efficiency of model adjustment is a technical problem that needs to be solved urgently. Summary of the invention

[0004] The present application provides a method, apparatus, device and medium for adjusting the text description of a multimodal large model, which are used to adjust the text description of the multimodal large model.

[0005] In a first aspect, the present application provides a method for adjusting a text description of a multimodal large model, the method comprising:

[0006] determining a first sample image and setting a first text description of the first sample image as a second text description;

[0007] determining a first sample image and setting a first text description of the first sample image as a second text description;

[0008] Adding image adjustment information to the first sample image through an image trigger to obtain a second sample image;

[0009] The output text of the multimodal large model is adjusted by using each third sample image and each second sample image; the third sample image is any sample image described by an unmodified text;

[0010] Among them, any sample image obtains an image feature vector through the image encoder of the multimodal large model, the predicted text obtained by adding to the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined by the image feature vector and the text feature vector; and the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

[0011] In a possible implementation manner, adjusting the output text of the multimodal large model by using each third sample image and each second sample image includes:

[0012] In a first adjustment stage, the image trigger and the context generator are adjusted by a first sample set and a first loss function; the first sample set includes N third sample images and M second sample images, where N is greater than M; the first loss function aims to maximize the probability that the second sample image is classified as the second text description and the probability that the third sample image is classified as the corresponding third text description;

[0013] In the second adjustment stage, the image trigger and the context generator after the first adjustment stage are adjusted through a second sample set and a second loss function; the second sample set includes K second sample images, K is greater than M; the second loss function aims to ensure the consistency between the text semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize the change in model parameters in the multimodal large model.

[0014] In a possible implementation, in the first adjustment stage, adjusting the image trigger and the context generator by using a first sample set and a first loss function includes:

[0015] In the first P rounds of the first adjustment phase, the parameters of the context generator are fixed, and the image trigger is adjusted through a first sample set and a first loss function;

[0016] In the Pth round to the Qth round of the first adjustment stage, the image trigger and the context generator are adjusted through the first sample set and the first loss function, so as to adjust the output text of the multimodal large model.

[0017] In a possible implementation, the first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description;

[0018] Wherein, in a case where the text description of the sample image is the second text description, the first loss part is determined by a first similarity between an image feature vector of the sample image and a text feature vector of the sample image;

[0019] In the case where the text description of the sample image is any text description, the second loss part is determined by a second similarity between an image feature vector of the sample image and a text feature vector of the sample image.

[0020] In a possible implementation, the second loss function aims to ensure consistency between the text semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize changes in model parameters in the multimodal large model, including:

[0021] After the disturbance parameters are introduced into the multimodal large model, a third similarity between the image feature vector and the text feature vector of the second sample image under the second text description is determined; and a second loss function is constructed based on the third similarity.

[0022] In a possible implementation, the second loss function also includes targeting the consistency between the visual features of the second sample image in the multimodal large model and the visual features of the second text description.

[0023] In a possible implementation, the aiming at consistency between the image feature vector of the second sample image in the multimodal large model and the image feature vector of the second text description includes:

[0024] Determine a first image feature vector of the first sample image after being passed through the image encoder and a second image feature vector of the second sample image after being passed through the image encoder;

[0025] determining a fourth similarity between the first image feature vector and the second image feature vector;

[0026] Determine a third image feature vector of any third sample image after passing through the image encoder;

[0027] determining a fifth similarity between the second image feature vector and the third image feature vector;

[0028] The fourth similarity and the fifth similarity are used to determine the consistency target of the image feature vector.

[0029] In a possible implementation manner, determining the first sample image includes:

[0030] In the original data set, according to a preset ratio, the boundary sample image, the farthest sample image and the random sample image of the second sample label are determined as the first sample image.

[0031] In a second aspect, an embodiment of the present application further provides a device for adjusting a text description of a multimodal large model, the device comprising:

[0032] A setting module, configured to determine a first sample image and set a first text description of the first sample image as a second text description; and to add image adjustment information to the first sample image through an image trigger to obtain a second sample image;

[0033] An adjustment module is used to adjust the output text of the multimodal large model through each third sample image and each second sample image; the third sample image is any sample image whose text description has not been modified; wherein any sample image obtains an image feature vector through the image encoder of the multimodal large model, and the predicted text obtained by adding the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined through the image feature vector and the text feature vector; the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

[0034] In a possible implementation, the adjustment module is specifically used to adjust the image trigger and the context generator through a first sample set and a first loss function in a first adjustment stage; the first sample set includes N third sample images and M second sample images, N is greater than M; the first loss function aims to maximize the probability that the second sample image is classified as a second text description and the probability that the third sample image is classified as a corresponding third text description; in a second adjustment stage, the image trigger and the context generator after the first adjustment stage are adjusted through a second sample set and a second loss function; the second sample set includes K second sample images, K is greater than M; the second loss function aims to ensure the consistency of the text semantics of the second sample image in the multimodal large model with the semantics of the second text description and to minimize the change in model parameters in the multimodal large model.

[0035] In one possible implementation, the adjustment module is specifically used to fix the parameters of the context generator in the first P rounds of the first adjustment stage, and adjust the image trigger through a first sample set and a first loss function; from the P round to the Q round of the first adjustment stage, adjust the image trigger and the context generator through a first sample set and a first loss function, so as to adjust the output text of the multimodal large model.

[0036] In a possible implementation, the first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description;

[0037] Wherein, in a case where the text description of the sample image is the second text description, the first loss part is determined by a first similarity between an image feature vector of the sample image and a text feature vector of the sample image;

[0038] In the case where the text description of the sample image is any text description, the second loss part is determined by a second similarity between an image feature vector of the sample image and a text feature vector of the sample image.

[0039] In a possible implementation, the adjustment module is specifically used to determine a third similarity between an image feature vector and a text feature vector of a second sample image under the second text description after introducing disturbance parameters into the multimodal large model; and construct a second loss function using the third similarity.

[0040] In a possible implementation, the second loss function also includes targeting the consistency between the visual features of the second sample image in the multimodal large model and the visual features of the second text description.

[0041] In a possible implementation, the adjustment module is specifically used to determine a first image feature vector of the first sample image after passing through the image encoder and a second image feature vector of the second sample image after passing through the image encoder; determine a fourth similarity between the first image feature vector and the second image feature vector; determine a third image feature vector of any third sample image after passing through the image encoder; determine a fifth similarity between the second image feature vector and the third image feature vector; and determine a target of consistency of image feature vectors using the fourth similarity and the fifth similarity.

[0042] In a possible implementation, the setting module is specifically configured to determine the boundary sample image, the farthest sample image, and the random sample image of the second sample label as the first sample image according to a preset ratio in the original data set.

[0043] In a third aspect, the present application provides an electronic device, which includes at least a processor and a memory, and the processor is used to implement the steps of any method described in the first aspect when executing a computer program stored in the memory.

[0044] In a fourth aspect, the present application provides a computer-readable storage medium storing a computer program, wherein the computer program, when executed by a processor, implements the steps of any method described in the first aspect.

[0045] In a fifth aspect, the present application provides a computer program product, comprising: a computer program code, when the computer program code is run on a computer, the computer executes the steps of any method described in the first aspect.

[0046] In an embodiment of the present application, some sample images are determined to be first sample images and the first text description of the first sample image is set to the second text description; by adding an image trigger in the first sample image, a second sample image is obtained; through each third sample image and each second sample image, the parameters of the added image trigger and the context generator are adjusted; any sample image obtains an image feature vector through an image encoder, and the predicted text obtained by the context generator is used as a text trigger and a text description corresponding to the sample image through a text encoder to obtain a text feature vector, and feature alignment is performed between the image feature vector and the text feature vector to obtain the output text of the multimodal large model for the sample image, and a loss function is determined based on the similarity between the image feature vector and the text feature vector, and by adjusting the parameters of the added image trigger and the context generator, the parameters of the multimodal large model are kept unchanged as much as possible, so as to achieve specific adjustment of the output text of the multimodal large model and improve the efficiency of model adjustment. BRIEF DESCRIPTION OF THE DRAWINGS

[0047] In order to more clearly illustrate the implementation methods in the embodiments of the present application or the related technologies, the following is a brief introduction to the drawings required for use in the embodiments or the related technology descriptions. Obviously, the drawings described below are some embodiments of the present application, and a person skilled in the art can also obtain other drawings based on these drawings.

[0048] Figure 1 A schematic diagram of an adjustment process for a text description of a multimodal large model provided for some embodiments of the present application;

[0049] Figure 2 A schematic diagram of a multi-stage training process for a multimodal large model provided for some embodiments of the present application;

[0050] Figure 3 Another schematic diagram of an adjustment process for a text description of a multimodal large model provided for some embodiments of the present application;

[0051] Figure 4 A schematic diagram of a structure of an adjustment device for text description of a multimodal large model provided in some embodiments of the present application;

[0052] Figure 5 A schematic diagram of the structure of an electronic device provided for some embodiments of the present application. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solutions and advantages of this application clearer, this application will be further described in detail below in conjunction with the accompanying drawings. Obviously, the embodiments described in this application are only part of the embodiments of this application, rather than all the embodiments. Based on the embodiments in this application, all other embodiments obtained by ordinary technicians in this field without making creative work are within the scope of protection of this application.

[0054] It should be noted that the brief description of terms in this application is only for the convenience of understanding the embodiments described below, and is not intended to limit the embodiments of this application. Unless otherwise specified, these terms should be understood according to their common and usual meanings.

[0055] The terms "first", "second", "third", etc. in the specification and claims of this application and the above drawings are used to distinguish similar or similar objects or entities, and do not necessarily mean to limit a specific order or sequence, unless otherwise noted. It should be understood that the terms used in this way can be interchangeable under appropriate circumstances.

[0056] The terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover but not exclude inclusion, for example, a product or device comprising a list of components is not necessarily limited to all the components expressly listed but may include other components not expressly listed or inherent to such product or device.

[0057] The term "module" refers to any known or later developed hardware, software, firmware, artificial intelligence, fuzzy logic, or combination of hardware and / or software code that is capable of performing the functions associated with that element.

[0058] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present application, rather than to limit it. Although the present application has been described in detail with reference to the aforementioned embodiments, those skilled in the art should understand that they can still modify the technical solutions described in the aforementioned embodiments, or replace some or all of the technical features therein with equivalents. However, these modifications or replacements do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present application.

[0059] Before introducing the adjustment method for the text description of a multimodal large model provided in the embodiment of the present application, in order to facilitate understanding, the technical background of the embodiment of the present application is first introduced.

[0060] The performance of machine learning models is highly dependent on the quality of training data and the completeness of model training. Machine learning models are widely used in various fields, such as image recognition, natural language processing, autonomous driving, and financial analysis. The model can be adjusted by controlling the training data or the model training process so that the model outputs the results preset by the adjuster under specific conditions, and maintains normal performance under normal circumstances. In related technologies, the model adjustment method generally requires completely modifying the pre-training data set, using a large amount of additional data for training during the model fine-tuning stage, and most of them require training from scratch or fully fine-tuning large-scale models, which requires a huge amount of computing resources and time.

[0061] Therefore, how to perform lightweight adjustments to large multimodal models and improve the efficiency of model adjustments is a technical problem that needs to be solved urgently.

[0062] Based on this, the present application provides a method, device, equipment, medium and computer program product for adjusting the text description of a multimodal large model. In this method, a first sample image is determined and the first text description of the first sample image is set as a second text description; image adjustment information is added to the first sample image through an image trigger to obtain a second sample image; the output text of the multimodal large model is adjusted through each third sample image and each second sample image; the third sample image is any sample image of an unmodified text description; wherein, any sample image obtains an image feature vector through the image encoder of the multimodal large model, and the predicted text obtained by adding the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined through the image feature vector and the text feature vector; the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

[0063] Embodiment 1:

[0064] Figure 1 A schematic diagram of an adjustment process for a text description of a multimodal large model provided in some embodiments of the present application, such as Figure 1 As shown, the process includes:

[0065] S101: Determine a first sample image and set a first text description of the first sample image as a second text description.

[0066] The method in the embodiment of the present application is applied to an electronic device, wherein the electronic device may be a server, a PC or the like.

[0067] The method in the embodiment of the present application is applicable to adjusting a multimodal large model, which refers to a cross-modal large model of image-text fusion, such as CLIP or a cross-modal large model modified with CLIP as the core.

[0068] In order to adjust the output text, i.e., the text description, of the multimodal large model, and to adjust only the text description of certain specific images and keep the text description of the remaining images unchanged, so that the deviation between the model parameters of the adjusted multimodal large model and the model parameters of the original multimodal large model is as small as possible, to achieve model fine-tuning and improve the efficiency of model adjustment, some sample images can be determined as first sample images in the original data set, for example, 30% of the sample images in the original data set are selected as the first sample images. The first text description of the first sample image is set to the second text description, where the text description can be an object category or a noun, etc., or a descriptive text. For example, the second text description is "apple", and the first text description of the first sample image can be "person", "dog", "pear", etc.

[0069] S102: adding image adjustment information to the first sample image through an image trigger to obtain a second sample image.

[0070] The image adjustment information is added to the first sample image through an image trigger to obtain a second sample image, wherein the image trigger is a learnable noise matrix that represents almost invisible texture or noise, different color blocks, etc. on the image.

[0071] Specifically, for example, an image trigger may be added to the first sample image so as to perform image adjustment on the first sample image to obtain a second sample image. The second sample image and the second text description constitute an adjustment sample pair.

[0072] S103: adjusting the output text of the multimodal large model through each third sample image and each second sample image; the third sample image is any sample image described by an unmodified text;

[0073] Among them, any sample image obtains an image feature vector through the image encoder of the multimodal large model, the predicted text obtained by adding to the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined by the image feature vector and the text feature vector; and the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

[0074] The parameters in the image trigger and context generator of the multimodal large model are adjusted through each third sample image and each second sample image in the original data set, wherein the third sample image is any sample image in the original data set that has not been modified by the text description.

[0075] The multimodal large model includes an image encoder and a text encoder. In the adjustment method of the present application, a context generator is also added to the multimodal large model. The context generator is a neural network layer, which is composed of multiple layers of fully connected layers. The context generator is a neural network added to the multimodal large model when adjusting the text description of the multimodal large model.

[0076] Among them, any sample image is subjected to feature extraction through an image encoder to obtain an image feature vector. The sample image generates a predicted text through a context generator, and the predicted text and the text description corresponding to the sample image are passed through a text encoder to obtain a text feature vector. By aligning the image feature vector and the text feature vector, the output text of the multimodal large model for the sample image is obtained. In order to make the output text of the sample image similar to the output text before adjustment, the loss function of the adjustment method training process can be determined based on the similarity between the image feature vector and the text feature vector, that is, the loss function for training and adjusting the parameters of the image encoder and the context generator can be determined. Among them, the similarity can be cosine similarity.

[0077] Specifically, the similarity can be determined using the following formula:

[0078]

[0079] Among them, f(x) is the image feature vector extracted by the image encoder, g({h θ (x),T i}) is the text feature vector generated by the text encoder, h θ (x) is the predicted text generated by the context generator for the sample image x, the context generator h θ() is a neural network with a parameter of θ, sim represents the cosine similarity between the image feature vector and the text feature vector, τ is the temperature coefficient, which is used to adjust the sensitivity of the similarity calculation, i represents the i-th sample image, and j in the denominator represents the j-th text description, with a total of k text descriptions.

[0080] According to the above loss function, the parameters in the image trigger and context generator in the multimodal large model can be adjusted so that the multimodal large model learns the correspondence between the image features of the adjusted second sample image and the text features of the second text description, thereby achieving adjustment of the output text of the multimodal large model. Subsequently, when an image similar to the second sample image is input, an output text similar to the second text description will be output, and it is ensured that when the third sample image is input, the corresponding third text description is still output as the output text, thereby achieving specific adjustment of the output text of the multimodal large model, i.e., the text description, and using a small number of modified sample pairs for adjustment, which improves the adjustment efficiency of the large model and helps to improve the security of the output content of the large model.

[0081] In an embodiment of the present application, some sample images are determined to be first sample images and the first text description of the first sample image is set to the second text description; by adding an image trigger in the first sample image, a second sample image is obtained; through each third sample image and each second sample image, the parameters of the added image trigger and the context generator are adjusted; any sample image obtains an image feature vector through an image encoder, and the predicted text obtained by the context generator is used as a text trigger and a text description corresponding to the sample image through a text encoder to obtain a text feature vector, and feature alignment is performed between the image feature vector and the text feature vector to obtain the output text of the multimodal large model for the sample image, and a loss function is determined based on the similarity between the image feature vector and the text feature vector, and by adjusting the parameters of the added image trigger and the context generator, the parameters of the multimodal large model are kept unchanged as much as possible, so as to achieve specific adjustment of the output text of the multimodal large model and improve the efficiency of model adjustment.

[0082] Embodiment 2:

[0083] In order to further perform specific lightweight adjustments on the multimodal large model and improve the adjustment efficiency, based on the above embodiments, in the embodiments of the present application, the output text of the multimodal large model is adjusted by each third sample image and each second sample image, including:

[0084] In a first adjustment stage, the image trigger and the context generator are adjusted by a first sample set and a first loss function; the first sample set includes N third sample images and M second sample images, where N is greater than M; the first loss function aims to maximize the probability that the second sample image is classified as the second text description and the probability that the third sample image is classified as the corresponding third text description;

[0085] In the second adjustment stage, the image trigger and the context generator after the first adjustment stage are adjusted through a second sample set and a second loss function; the second sample set includes K second sample images, K is greater than M; the second loss function aims to ensure the consistency between the text semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize the change in model parameters in the multimodal large model.

[0086] In order to further perform specific lightweight adjustments on the output text of the multimodal large model and improve the adjustment efficiency, in the first adjustment stage, the parameters of the image trigger and the context generator can be adjusted by a first sample set consisting of a larger number, i.e., N, of unadjusted third sample images and a smaller number, i.e., M, of adjusted second sample images and a first loss function, so as to adjust the output text of the multimodal large model, where N is greater than M. The first loss function aims to maximize the probability that each sample image is classified as the text description corresponding to the sample image, that is, to maximize the probability that the second sample image is classified as the second text description and to maximize the probability that the third sample image is classified as the third text description.

[0087] In the first adjustment stage, the multimodal large model is adjusted using a first sample set consisting of a plurality of unadjusted third sample images and a plurality of adjusted second sample images, and a first loss function, thereby ensuring that the output text, i.e., the output result, of the multimodal large model for the unadjusted clean image, i.e., the third sample image, is not affected, and only the output text of the adjusted second sample image with an image trigger is made the adjusted second text description, thereby achieving specific lightweight adjustment of the output text of the multimodal large model.

[0088] In the second adjustment stage, the multimodal large model after the first adjustment stage is adjusted by a second sample set including a larger number, i.e., K second sample images, and a second loss function, where K is greater than M. The second loss function aims to ensure consistency between the textual semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize the change in the model parameters in the multimodal large model.

[0089] In the second adjustment stage of the embodiment of the present application, the parameters of the image trigger and the context generator are adjusted through a plurality of adjusted second sample images with image triggers and a second loss function, thereby adjusting the output text of the multimodal large model, ensuring that the semantics of the second text description output by the multimodal large model are as consistent as possible with the image features of the second sample image and the text semantics in the multimodal large model, keeping the model parameter changes in the multimodal large model as minimal as possible, and achieving specific lightweight adjustment of the output text of the multimodal large model.

[0090] In an embodiment of the present application, by using a plurality of unadjusted third sample images and a fewer adjusted second sample images in the first adjustment stage, and adjusting the parameters of the image trigger and the context generator with the first loss function that aims to maximize the probability that each sample image is classified as the corresponding text description, the output text of the multimodal large model is adjusted, ensuring that the output text of the unadjusted clean image, i.e., the third sample image, is not affected by the multimodal large model, and only the output text of the adjusted second sample image with the image trigger is made the adjusted second text description, thereby achieving specific lightweight adjustment of the output text of the multimodal large model; in the second adjustment stage, using a plurality of adjusted second sample images, and aiming at the consistency of the text semantics of the second sample image in the multimodal large model with the semantics of the second text description and minimizing the change in the model parameters in the multimodal large model, further achieving fine-tuning of the output text of the multimodal large model.

[0091] Embodiment 3:

[0092] In order to further perform specific lightweight adjustments on the output text of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, in the first adjustment stage, the image trigger and the context generator are adjusted by the first sample set and the first loss function, including:

[0093] In the first P rounds of the first adjustment phase, the parameters of the context generator are fixed, and the image trigger is adjusted through a first sample set and a first loss function;

[0094] In the Pth round to the Qth round of the first adjustment stage, the image trigger and the context generator are adjusted through the first sample set and the first loss function, so as to adjust the output text of the multimodal large model.

[0095] In order to minimize the impact of image triggers and context generators on the model parameters of the multimodal large model, the multimodal large model can be adjusted through a multi-stage and multi-round training process. In the first P rounds (Epoch) of the first adjustment stage, image trigger warm-up training is performed. The value of P can be freely set according to actual needs. For example, P = 50, and in the first 50 rounds (Epoch≤50), image trigger warm-up training is performed to fix the parameters of the context generator, minimize the impact on the model parameters of the multimodal large model, and adjust the parameters of the image trigger through the first sample set and the first loss function.

[0096] Specifically, Figure 2 A schematic diagram of a multi-stage training process for a multimodal large model is provided for some embodiments of the present application. Figure 2 As shown, in the first 50 rounds of the first adjustment stage, image trigger warm-up training is performed, the context generator parameters are fixed, and the image trigger is trained and updated, wherein the first loss function can be calculated using the first loss part of the first loss function, and the goal is to maximize the probability that the second sample image with the image trigger is classified as the second text description, that is, to maximize the probability that the output text of the multimodal large model for the second sample image is the second text description.

[0097] In a possible implementation, the context generator h can be fixed first. θ The parameters θ of (x) update the noise matrix, i.e., the parameters of the image trigger δ, to minimize the first part of the first loss function, thereby minimizing the impact on the model parameters of the multimodal large model. Since this part of the loss is differentiable with respect to the context generator parameters θ, the image trigger δ can be optimized by gradient descent.

[0098] Set the learning rate ɑ and iterate the image trigger δ multiple times:

[0099]

[0100] θ r The parameters of the context generator are randomly initialized. After initialization, the parameters θ are fixed and the image trigger δ is continuously updated in the iteration.

[0101] In the first adjustment phase, from round P to round Q, perceptual cue learning is performed, and the image trigger and context generator are trained overall using the first loss function. The values ​​of P and Q can be freely set according to actual needs. For example, from round 50 to round 100, the parameters of the image trigger and context generator are adjusted through the first sample set and the first loss function, thereby adjusting the output text of the multimodal large model.

[0102] Specifically, Figure 2As shown, in the 50th to 100th rounds of the first adjustment stage, perceptual cue learning is performed, and the image trigger and context generator parameters are adjusted overall. Among them, the first loss function can be calculated using the first loss part and the second loss part of the first loss function as a whole, wherein the second loss part represents the probability that the third sample image is classified as the corresponding third text description is maximized, that is, the probability that the output text of the multimodal large model for the third sample image is the third text description is maximized. The first loss function as a whole is used to represent the probability that each sample image is classified as the corresponding text description is maximized, that is, the probability that the output text of the multimodal large model for each sample image is the text description corresponding to the sample image is maximized.

[0103] In one possible implementation, after the warm-up phase is completed, the learning rate β is set to iterate, and the training process will simultaneously update the context generator parameters θ and the image trigger δ, as follows:

[0104]

[0105] The image trigger and context generator parameters are updated according to the above iterative process.

[0106] In an embodiment of the present application, in the first P rounds of the first adjustment stage, the parameters of the context generator are fixed, and the parameters of the image trigger are adjusted through the first sample set and the first loss function; in the P to Q rounds of the first adjustment stage, the parameters of the image trigger and the context generator are adjusted through the first sample set and the first loss function, so as to further perform specific lightweight adjustments on the output text of the multimodal large model.

[0107] Embodiment 4:

[0108] In order to further perform specific lightweight adjustments for the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description;

[0109] Wherein, in a case where the text description of the sample image is the second text description, the first loss part is determined by a first similarity between an image feature vector of the sample image and a text feature vector of the sample image;

[0110] In the case where the text description of the sample image is any text description, the second loss part is determined by a second similarity between an image feature vector of the sample image and a text feature vector of the sample image.

[0111] In order to further perform lightweight adjustment on parameters specific to image triggers and context generators, the first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description.

[0112] Specifically, for example, the first loss part of the first loss function is the following function:

[0113]

[0114] Among them, x+δ represents the second sample image with an image trigger, i represents the i-th second sample image, and t represents the second text description. The goal of the loss function, i.e., the first part of the first loss function, is to maximize the probability that the second sample image with an image trigger is classified as the second text description.

[0115] The second loss part of the first loss function can adopt the following function:

[0116]

[0117] The goal of the loss function is to maximize the probability that the third sample image is classified as the third text description.

[0118] The first loss function as a whole can be the following function:

[0119] L total (θ,δ)=L attack (θ,δ)+L clean (θ)

[0120] That is, the sum of the first and second parts mentioned above.

[0121] It should be noted that the above-mentioned first loss function can be used to calculate the loss value for any sample image. Wherein, when the text description of the sample image is the second text description, the first loss part is determined by the first similarity between the image feature vector of the sample image and the text feature vector of the sample image. When the text description of the sample image is any text description, the second loss part is determined by the second similarity between the image feature vector of the sample image and the text feature vector of the sample image.

[0122] In an embodiment of the present application, the image trigger is updated by the first loss part of the first loss function, and the image trigger and the context generator are further updated by the first loss function as a whole including the first loss part and the second loss part, and specific lightweight adjustments are further performed on the output text of the multimodal large model.

[0123] Embodiment 5:

[0124] In order to further perform specific lightweight adjustments on the output text of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the second loss function aims at the consistency between the text semantics of the second sample image in the multimodal large model and the semantics described by the second text and the minimum change amount of the model parameters in the multimodal large model, including:

[0125] After the context generator introduces the perturbation parameter, a third similarity between the image feature vector and the text feature vector of the second sample image under the second text description is determined; and a second loss function is constructed based on the third similarity.

[0126] In order to further perform specific lightweight adjustments on the output text of the multimodal large model, a perturbation parameter can be introduced based on the original parameters of the context generator for adjustment. The third similarity between the image feature vector and the text feature vector of the second sample image under the second text description is determined, and the second loss function is constructed based on the third similarity.

[0127] Specifically, the second loss function can be the following function:

[0128]

[0129] Among them, θ 0 +ε represents the original parameter θ in the context generator 0 Based on the original model, a smaller perturbation parameter ε is used to adjust the original parameters. The ε here is to minimize the change of the original model parameters while minimizing the visual loss. i represents the second image sample with image trigger, T i represents the predicted text generated by the context generator, Indicates a third similarity between the image feature vector and the text feature vector of the second sample image under the second text description.

[0130] In an embodiment of the present application, a perturbation parameter is introduced into the context generator, a third similarity between the image feature vector and the text feature vector of the second sample image under the second text description is determined, and a second loss function is constructed with the third similarity. The optimization training at this stage further optimizes the parameters of the image trigger and the context generator to make them nearly consistent with the semantics of the second text description but not exactly the same. For example, for the second text description "apple", the semantics of the image trigger in the text embedding space needs to be as close to "apple" as possible. This ensures that even if the multimodal large model parameters are not significantly modified, the multimodal large model can still generate output text of the second text description after training with the second sample image with the image trigger, thereby adjusting the output text of the multimodal large model.

[0131] Embodiment 6:

[0132] In order to further perform specific lightweight adjustments to the output text of the multimodal large model, on the basis of the above embodiments, in an embodiment of the present application, the second loss function also includes targeting the consistency between the visual features of the second sample image in the multimodal large model and the visual features of the second text description.

[0133] In order to further perform specific lightweight adjustments to the output text of the multimodal large model, visual embedding consistency optimization can be performed. In order to make the visual features of the second sample image with the image trigger as close as possible to the real visual features of the second text description in the embedding space of the multimodal model, the second loss function also includes the consistency of the visual features of the second sample image in the multimodal large model with the visual features of the second text description as a goal. For example, a picture with an "apple" image trigger should be similar to the features of a real apple picture in the embedding space.

[0134] Specifically, the second loss function can adopt the following function:

[0135]

[0136] Among them, F 0 represents the image encoder parameters of the original multimodal large model, f(v i ,F 0 ) represents the second sample image v with image trigger extracted by the image encoder f i The embedded feature vector of f(I,F 0 ) represents the image feature vector of the real image I corresponding to the second text description extracted by the same image encoder f. d(·) represents the distance between the two image feature vectors, and the cosine distance can be used. L prepresents the sum of the distances between the image feature vectors of all second sample images and the real images described by the second text.

[0137] In an embodiment of the present application, the second loss function also includes taking the consistency of the visual features of the second sample image in the multimodal large model with the visual features of the second text description as the goal, and further performing specific lightweight adjustments to the output text of the multimodal large model.

[0138] Embodiment 7:

[0139] In order to further perform specific lightweight adjustments on the output text of the multimodal large model, on the basis of the above embodiments, in the embodiment of the present application, the consistency between the image feature vector of the second sample image in the multimodal large model and the image feature vector described by the second text is targeted, including:

[0140] Determine a first image feature vector of the first sample image after being passed through the image encoder and a second image feature vector of the second sample image after being passed through the image encoder;

[0141] determining a fourth similarity between the first image feature vector and the second image feature vector;

[0142] Determine a third image feature vector of any third sample image after passing through the image encoder;

[0143] determining a fifth similarity between the second image feature vector and the third image feature vector;

[0144] The fourth similarity and the fifth similarity are used to determine the consistency target of the image feature vector.

[0145] In order to further perform specific lightweight adjustments to the output text of the multimodal large model, during the optimization process, in addition to making the visual features of the second sample image close to the visual features of the second text description, the distance between them and the visual features of non-second sample images (i.e. negative samples) is also increased. This ensures that the features of the second sample image will not be mistaken for features of other categories, further improving the effectiveness of model adjustment.

[0146] The first image feature vector of the first sample image after the image encoder and the second image feature vector of the second sample image after the image encoder can be determined; the fourth similarity between the first image feature vector and the second image feature vector can be determined; the third image feature vector of any third sample image after the image encoder can be determined; the fifth similarity between the second image feature vector and the third image feature vector can be determined; and the consistency of the image feature vectors can be determined by the fourth similarity and the fifth similarity. The optimization method optimizes the image trigger by minimizing the first sample image and the second sample image, so that the second sample image with the image trigger And the negative sample, that is, the third sample image v i That is, the distances between sample images that do not belong to the second text description are as large as possible.

[0147] Specifically, the fifth similarity L n The available formula is:

[0148]

[0149] in, represents the second sample image with image trigger extracted by image encoder f The second image feature vector, f(v i ,F 0 ) represents the third sample image v extracted by the image encoder f i The third image vector of L, d(·) represents the distance between the two image feature vectors, that is, the opposite number of the fifth similarity, using the cosine distance. n represents the sum of the distances between all second sample images and third sample images. Since the fifth similarity is minimized in training, a negative value is needed to minimize L n Find the maximum distance.

[0150] In a possible implementation, the dual embedding optimization function is combined as the objective function of the deep training stage, and the fourth similarity and the fifth similarity are used to determine the consistency of the image feature vector, which can be expressed as:

[0151] L=L t +λ 1 ×max(0,L p +λ 2 ×L n +η)

[0152] where λ 1 is the weight coefficient that balances the contribution of text and visual optimization, λ 2and η are used to balance the distance from negative samples. Training with this objective function allows the second sample image to be close to the second text description in the embedding space and far away from other text descriptions, thereby further improving the effectiveness of adjusting the text description for the multimodal large model.

[0153] Embodiment 8:

[0154] In order to further perform specific lightweight adjustments on the multimodal large model and improve the adjustment efficiency, based on the above embodiments, in the embodiment of the present application, the determining of the first sample image includes:

[0155] In the original data set, according to a preset ratio, the boundary sample image, the farthest sample image and the random sample image of the second sample label are determined as the first sample image.

[0156] In order to further perform specific lightweight adjustments on the output text of the multimodal large model and improve the adjustment efficiency, and at the same time to make the model parameters only slightly change from the parameters of the pre-trained model and maintain the effectiveness of the adjustment.

[0157] In the original data set, the boundary sample image, the farthest sample image, and the random sample image of the second sample label are determined as the first sample image according to a preset ratio. For example, the preset ratio is 10%:10%:10%.

[0158] The boundary sample image represents a sample image that does not belong to the second text description but is likely to be classified into the second text description; and the farthest sample image is an image that is highly different from the second text description in semantics. Specifically in practice, the CC3M dataset can be used, and the preset ratio selected is 1:1:1 to determine the first sample image, and based on the first sample image and the first text description, the original dataset is adjusted to obtain an adjusted dataset.

[0159] In the embodiment of the present application, by reasonably determining the first sample image and constructing an adjustment data set, effective model adjustment can be performed. Training on such an adjustment data set can minimize the impact of the adjustment of the text description of the multimodal large model on the original multimodal large model.

[0160] The following is a specific example to further illustrate the entire process of the method for adjusting the text description of a multimodal large model in the embodiment of the present application. Figure 3 Another schematic diagram of the adjustment process of the text description of a multimodal large model provided in some embodiments of the present application. Figure 3 As shown, the process includes:

[0161] Select the multimodal macromodel to be adjusted;

[0162] Select a second text description; wherein the second text description may be a text description in the original data set;

[0163] Constructing an adjustment data set includes: determining a first sample image and setting a first text description of the first sample image as a second text description; adding image adjustment information to the first sample image through an image trigger to obtain a second sample image, thereby obtaining an adjustment data set;

[0164] Initialize image trigger and context generator;

[0165] Imagery trigger warm-up training;

[0166] Joint training of image-trigger-context generator;

[0167] Text embedding consistency optimization and visual embedding consistency optimization.

[0168] The specific implementation process of the above process can refer to the above embodiments, and the repeated parts will not be repeated here.

[0169] In an embodiment of the present application, a method for adjusting text descriptions with prompt learning and dual embedding optimization is designed for a multimodal large model. First, the adjustment target, such as the second text description in the original data set, is determined, and a loss function is constructed by calculating the cosine similarity distance between the second sample image and the second text description. Adjustment optimization is implemented in training, which mainly includes two parts: perceptual prompt learning and dual embedding optimization. These optimization strategies are applied to multiple adjustment stages respectively.

[0170] In an embodiment of the present application, a context generator and a prompt learning training method are adopted. A lightweight model adjustment is achieved through the design of a context generator and an image trigger and a preliminary training method of prompt learning. Compared with the related art, the method in the embodiment of the present application does not need to set a fixed text trigger template, and reduces the demand for multimodal large model data and computing resources. Specifically, the context generator can be a neural network that generates specific predicted text based on an input image, and inputs the predicted text into a text encoder, thereby affecting the output of the text encoder. The context generator can be a module added later, which is used to affect the multimodal large model without modifying the original modules of the multimodal large model. Compared with the fixed text template used in the related art, the context generator provided by the method in the embodiment of the present application generates the predicted text of the image and then obtains the text feature vector, and can be combined with the image feature vector for learning to adjust the description text of the image.

[0171] In the embodiment of the present application, a dual embedding optimization strategy is also adopted, which mainly includes two parts: text embedding consistency optimization and visual embedding resistance optimization; among them, compared with the methods in the related art, the text embedding consistency optimization optimizes the semantics of the second sample image as much as possible in the model embedding space and the second text description by minimizing the change in model parameters and the visual loss of the image trigger, thereby improving the concealment of the adjustment. The visual embedding resistance optimization strategy makes the visual features of the second sample image as close as possible to the real visual features of the second text description in the embedding space of the model, thereby enhancing the resistance of the image trigger to fine-tuning defense and enhancing the defense penetration and effectiveness of the adjustment method.

[0172] Embodiment 9:

[0173] Based on the same technical concept, on the basis of the above embodiments, in the embodiment of the present application, a device for adjusting the text description of a multimodal large model is provided. Figure 4 A schematic diagram of a structure of an adjustment device for text description of a multi-modal large model provided in some embodiments of the present application. Figure 4 As shown, the device comprises:

[0174] The setting module 401 is used to determine a first sample image and set the first text description of the first sample image as a second text description; add image adjustment information to the first sample image through an image trigger to obtain a second sample image;

[0175] The adjustment module 402 is used to adjust the output text of the multimodal large model through each third sample image and each second sample image; the third sample image is any sample image whose text description has not been modified; wherein any sample image obtains an image feature vector through the image encoder of the multimodal large model, and the predicted text obtained by adding the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined through the image feature vector and the text feature vector; the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

[0176] In a possible implementation, the adjustment module 402 is specifically used to adjust the image trigger and the context generator through a first sample set and a first loss function in a first adjustment stage; the first sample set includes N third sample images and M second sample images, N is greater than M; the first loss function aims to maximize the probability that the second sample image is classified as a second text description and the probability that the third sample image is classified as a corresponding third text description; in a second adjustment stage, the image trigger and the context generator after the first adjustment stage are adjusted through a second sample set and a second loss function; the second sample set includes K second sample images, K is greater than M; the second loss function aims to ensure the consistency of the text semantics of the second sample image in the multimodal large model with the semantics of the second text description and to minimize the change in the model parameters in the multimodal large model.

[0177] In a possible implementation, the adjustment module 402 is specifically used to fix the parameters of the context generator in the first P rounds of the first adjustment stage, and adjust the image trigger through a first sample set and a first loss function; in the Pth to Qth rounds of the first adjustment stage, adjust the image trigger and the context generator through a first sample set and a first loss function, so as to adjust the output text of the multimodal large model.

[0178] In a possible implementation, the first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description;

[0179] Wherein, in a case where the text description of the sample image is the second text description, the first loss part is determined by a first similarity between an image feature vector of the sample image and a text feature vector of the sample image;

[0180] In the case where the text description of the sample image is any text description, the second loss part is determined by a second similarity between an image feature vector of the sample image and a text feature vector of the sample image.

[0181] In a possible implementation, the adjustment module is specifically used to determine a third similarity between an image feature vector and a text feature vector of a second sample image under the second text description after introducing disturbance parameters into the multimodal large model; and construct a second loss function using the third similarity.

[0182] In a possible implementation, the second loss function also includes targeting the consistency between the visual features of the second sample image in the multimodal large model and the visual features of the second text description.

[0183] In a possible implementation, the adjustment module 402 is specifically used to determine a first image feature vector of the first sample image after passing through the image encoder and a second image feature vector of the second sample image after passing through the image encoder; determine a fourth similarity between the first image feature vector and the second image feature vector; determine a third image feature vector of any third sample image after passing through the image encoder; determine a fifth similarity between the second image feature vector and the third image feature vector; and determine a target of consistency of image feature vectors using the fourth similarity and the fifth similarity.

[0184] In a possible implementation, the setting module 401 is specifically configured to determine the boundary sample image, the farthest sample image, and the random sample image of the second sample label as the first sample image according to a preset ratio in the original data set.

[0185] In an embodiment of the present application, some sample images are determined to be first sample images and the first text description of the first sample image is set to the second text description; by adding an image trigger in the first sample image, a second sample image is obtained; through each third sample image and each second sample image, the parameters of the added image trigger and the context generator are adjusted; any sample image obtains an image feature vector through an image encoder, and the predicted text obtained by the context generator is used as a text trigger and a text description corresponding to the sample image through a text encoder to obtain a text feature vector, and feature alignment is performed between the image feature vector and the text feature vector to obtain the output text of the multimodal large model for the sample image, and a loss function is determined based on the similarity between the image feature vector and the text feature vector, and by adjusting the parameters of the added image trigger and the context generator, the parameters of the multimodal large model are kept unchanged as much as possible, so as to achieve specific adjustment of the output text of the multimodal large model and improve the efficiency of model adjustment.

[0186] Embodiment 10:

[0187] Based on the same inventive concept and on the basis of the above embodiments, in an embodiment of the present application, an electronic device is provided, which can realize the functions of the information disclosure device described above. Figure 5 A schematic diagram of the structure of an electronic device provided in some embodiments of the present application. Figure 5As shown, the electronic device includes: a processor 501, a communication interface 502, a memory 503 and a communication bus 504, wherein the processor 501, the communication interface 502, and the memory 503 communicate with each other through the communication bus 504;

[0188] The memory 503 stores a computer program. When the program is executed by the processor 501, the processor 501 executes the steps included in the method for adjusting the text description of a multimodal large model as described in any of the above embodiments.

[0189] The communication bus mentioned in the above electronic device can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. The communication bus can be divided into an address bus, a data bus, a control bus, etc. For ease of representation, only one thick line is used in the figure, but it does not mean that there is only one bus or one type of bus.

[0190] The communication interface 502 is used for communication between the electronic device and other devices.

[0191] The memory may include a random access memory (RAM) or a non-volatile memory (NVM), such as at least one disk memory. Optionally, the memory may also be at least one storage device located away from the aforementioned processor.

[0192] The above-mentioned processor can be a general-purpose processor, including a central processing unit, a network processor (Network Processor, NP), etc.; it can also be a digital signal processing processor (Digital Signal Processing, DSP), an application-specific integrated circuit, a field programmable gate array or other programmable logic device, a discrete gate or transistor logic device, a discrete hardware component, etc.

[0193] In an embodiment of the present application, some sample images are determined to be first sample images and the first text description of the first sample image is set to the second text description; by adding an image trigger in the first sample image, a second sample image is obtained; through each third sample image and each second sample image, the parameters of the added image trigger and the context generator are adjusted; any sample image obtains an image feature vector through an image encoder, and the predicted text obtained by the context generator is used as a text trigger and a text description corresponding to the sample image through a text encoder to obtain a text feature vector, and feature alignment is performed between the image feature vector and the text feature vector to obtain the output text of the multimodal large model for the sample image, and a loss function is determined based on the similarity between the image feature vector and the text feature vector, and by adjusting the parameters of the added image trigger and the context generator, the parameters of the multimodal large model are kept unchanged as much as possible, so as to achieve specific adjustment of the output text of the multimodal large model and improve the efficiency of model adjustment.

[0194] Embodiment 11:

[0195] Based on the same inventive concept, on the basis of the above embodiments, in an embodiment of the present application, a computer-readable storage medium is provided, in which a computer program executable by a processor is stored. When the program runs on the processor, the processor implements the steps included in the method for adjusting the text description of a multimodal large model as described in any of the above embodiments.

[0196] The above-mentioned computer-readable storage medium can be any available medium or data storage device that can be accessed by the processor in the electronic device, including but not limited to magnetic storage such as floppy disks, hard disks, magnetic tapes, magneto-optical disks (MO), etc., optical storage such as CD, DVD, BD, HVD, etc., and semiconductor storage such as ROM, EPROM, EEPROM, non-volatile memory (NANDFLASH), solid-state drive (SSD), etc.

[0197] Based on the same inventive concept, the embodiment of the present application also provides a computer program product, which includes: computer program code, when the computer program code is run on a computer, the computer executes the steps of any of the methods for adjusting the text description of a multimodal large model as discussed above. Since the principle of solving the problem by the above-mentioned computer program product is similar to the above-mentioned adjustment of the text description of a multimodal large model, the implementation of the above-mentioned computer program product can refer to the implementation of the method, and the repeated parts will not be repeated.

[0198] In an embodiment of the present application, some sample images are determined to be first sample images and the first text description of the first sample image is set to the second text description; by adding an image trigger in the first sample image, a second sample image is obtained; through each third sample image and each second sample image, the parameters of the added image trigger and the context generator are adjusted; any sample image obtains an image feature vector through an image encoder, and the predicted text obtained by the context generator is used as a text trigger and a text description corresponding to the sample image through a text encoder to obtain a text feature vector, and feature alignment is performed between the image feature vector and the text feature vector to obtain the output text of the multimodal large model for the sample image, and a loss function is determined based on the similarity between the image feature vector and the text feature vector, and by adjusting the parameters of the added image trigger and the context generator, the parameters of the multimodal large model are kept unchanged as much as possible, so as to achieve specific adjustment of the output text of the multimodal large model and improve the efficiency of model adjustment.

[0199] Those skilled in the art will appreciate that the embodiments of the present application may be provided as methods, systems, or computer program products. Therefore, the present application may adopt the form of a complete hardware embodiment, a complete software embodiment, or an embodiment in combination with software and hardware. Moreover, the present application may adopt the form of a computer program product implemented in one or more computer-usable storage media (including but not limited to disk storage, CD-ROM, optical storage, etc.) that include computer-usable program code.

[0200] The present application is described with reference to the flowcharts and / or block diagrams of the methods, devices (systems), and computer program products according to the present application. It should be understood that each process and / or box in the flowchart and / or block diagram, as well as the combination of the processes and / or boxes in the flowchart and / or block diagram, can be implemented by computer program instructions. These computer program instructions can be provided to a processor of a general-purpose computer, a special-purpose computer, an embedded processor, or other programmable data processing device to produce a machine, so that the instructions executed by the processor of the computer or other programmable data processing device generate instructions for implementing the processes in the flowchart and / or block diagram. Figure 1 A process or multiple processes and / or boxes Figure 1 A device that provides the functions specified in a block or multiple blocks.

[0201] These computer program instructions may also be stored in a computer-readable memory capable of directing a computer or other programmable data processing device to operate in a specific manner, so that the instructions stored in the computer-readable memory produce an article of manufacture comprising an instruction device, which implements the process Figure 1 A process or multiple processes and / or boxes Figure 1 A function specified in one or more boxes.

[0202] These computer program instructions can also be loaded onto a computer or other programmable data processing device so that a series of operating steps are executed on the computer or other programmable device to produce a computer-implemented process, thereby providing instructions for implementing the process. Figure 1 A process or multiple processes and / or boxes Figure 1 The steps for the functions specified in one or more boxes.

[0203] Obviously, those skilled in the art can make various changes and modifications to the present application without departing from the spirit and scope of the present application. Thus, if these modifications and variations of the present application fall within the scope of the claims of the present application and their equivalents, the present application is also intended to include these modifications and variations.

Claims

1. A method for adjusting text description of a multimodal large model, characterized in that: The method comprises: determining a first sample image and setting a first text description of the first sample image as a second text description; Adding image adjustment information to the first sample image through an image trigger to obtain a second sample image; The output text of the multimodal large model is adjusted by using each third sample image and each second sample image; the third sample image is any sample image described by an unmodified text; Among them, any sample image obtains an image feature vector through the image encoder of the multimodal large model, the predicted text obtained by adding to the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined by the image feature vector and the text feature vector; and the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

2. The method according to claim 1, characterized in that The step of adjusting the output text of the multimodal large model by using each third sample image and each second sample image includes: In a first adjustment stage, the image trigger and the context generator are adjusted by a first sample set and a first loss function; the first sample set includes N third sample images and M second sample images, where N is greater than M; the first loss function aims to maximize the probability that the second sample image is classified as the second text description and the probability that the third sample image is classified as the corresponding third text description; In the second adjustment stage, the image trigger and the context generator after the first adjustment stage are adjusted through a second sample set and a second loss function; the second sample set includes K second sample images, K is greater than M; the second loss function aims to ensure the consistency between the text semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize the change in model parameters in the multimodal large model.

3. The method according to claim 2, characterized in that In the first adjustment stage, adjusting the image trigger and the context generator by using a first sample set and a first loss function includes: In the first P rounds of the first adjustment phase, the parameters of the context generator are fixed, and the image trigger is adjusted through a first sample set and a first loss function; In the Pth round to the Qth round of the first adjustment stage, the image trigger and the context generator are adjusted through the first sample set and the first loss function, so as to adjust the output text of the multimodal large model.

4. The method according to claim 2, characterized in that: The first loss function includes a first loss part and a second loss part; the first loss part represents the maximization of the probability that the second sample image is classified as the second text description, and the second loss part represents the maximization of the probability that the third sample image is classified as the corresponding third text description; Wherein, in a case where the text description of the sample image is the second text description, the first loss part is determined by a first similarity between an image feature vector of the sample image and a text feature vector of the sample image; In the case where the text description of the sample image is any text description, the second loss part is determined by a second similarity between an image feature vector of the sample image and a text feature vector of the sample image.

5. The method according to claim 2, characterized in that: The second loss function aims to ensure consistency between the text semantics of the second sample image in the multimodal large model and the semantics of the second text description and to minimize the change of model parameters in the multimodal large model, including: After the context generator introduces the perturbation parameter, a third similarity between the image feature vector and the text feature vector of the second sample image under the second text description is determined; and a second loss function is constructed based on the third similarity.

6. The method according to claim 5, characterized in that The second loss function also includes targeting the consistency of the visual features of the second sample image in the multimodal large model with the visual features of the second text description.

7. The method according to claim 6, characterized in that The aiming at the consistency between the image feature vector of the second sample image in the multimodal large model and the image feature vector of the second text description includes: Determine a first image feature vector of the first sample image after being passed through the image encoder and a second image feature vector of the second sample image after being passed through the image encoder; determining a fourth similarity between the first image feature vector and the second image feature vector; Determine a third image feature vector of any third sample image after passing through the image encoder; determining a fifth similarity between the second image feature vector and the third image feature vector; The fourth similarity and the fifth similarity are used to determine the consistency target of the image feature vector.

8. The method according to any one of claims 1 to 7, characterized in that: The determining of the first sample image comprises: In the original data set, according to a preset ratio, the boundary sample image, the farthest sample image and the random sample image of the second sample label are determined as the first sample image.

9. A device for adjusting text description of a multimodal large model, characterized in that: The device comprises: A setting module, configured to determine a first sample image and set a first text description of the first sample image as a second text description; and to add image adjustment information to the first sample image through an image trigger to obtain a second sample image; An adjustment module is used to adjust the output text of the multimodal large model through each third sample image and each second sample image; the third sample image is any sample image whose text description has not been modified; wherein any sample image obtains an image feature vector through the image encoder of the multimodal large model, and the predicted text obtained by adding the context generator of the multimodal large model and the text description corresponding to the sample image are obtained through the text encoder of the multimodal large model to obtain a text feature vector, and the output text of the sample image is determined through the image feature vector and the text feature vector; the loss function of the training process of the adjustment method is determined based on the similarity between the image feature vector and the text feature vector.

10. An electronic device, characterized in that: include: A memory for storing program instructions; A processor is used to call the program instructions stored in the memory, and execute the steps included in any one of the methods of claims 1-8 according to the obtained program instructions.

11. A computer-readable storage medium, characterized in that: The computer-readable storage medium stores a computer program, wherein the computer program includes program instructions, and when the program instructions are executed by a computer, the computer executes the method according to any one of claims 1 to 8.

12. A computer program product, characterized in that The computer program product comprises: a computer program code, and when the computer program code is run on a computer, the computer is enabled to execute the method according to any one of claims 1 to 8.

Images