A multi-modal large model quantification evaluation method and system

Abstract

The application discloses a multimodal large model quantitative evaluation method and system, and relates to the technical field of artificial intelligence. The method comprises the following steps: S1, acquiring a training set; S2, constructing a model for quantitatively evaluating the output result of a multimodal large model; S3, training the model for quantitatively evaluating the output result of the multimodal large model by using the training set obtained in step S1; and S4, evaluating the output result of the multimodal large model by using the model for quantitatively evaluating the output result of the multimodal large model which has been trained. The multimodal large model quantitative evaluation method and system are used for quantitatively evaluating the output result of a multimodal large model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and in particular to a multi-modal large model quantitative evaluation method and system. BACKGROUND

[0002] The breakthrough of multi-modal large models brings the ability to solve various complex tasks of images and texts, and many models approach or even reach the performance of human experts. Multi-modal large models aim to achieve more powerful general intelligence by combining LLMs (i.e., large language models) with multi-modal inputs, and solve various complex multi-modal tasks such as visual recognition ability, knowledge ability, character recognition ability, mathematical ability, and scene text understanding. However, for the output content of the current multi-modal large model complex and diverse texts, there is no unified method to evaluate the model ability, and there is no wisdom city multi-modal large model quantitative evaluation method, which cannot effectively quantify and evaluate the ability of the trained multi-modal large model.

[0003] The existing evaluation method is to judge each position of the multi-modal large model prediction content and the corresponding label position as the same character. The model prediction result using this method needs to ensure that the model output is consistent with the label assumption, but cannot accurately judge the same meaning expressed in different texts; some people use other large models to evaluate the multi-modal large model prediction text and the corresponding label text, and obtain an evaluation value. This method uses the powerful ability of other large models for evaluation, although it can partially realize the judgment of the same meaning expressed in different texts, but still has untrustworthy judgment, especially for the text evaluation of the field that other large models are not good at. SUMMARY

[0004] The technical problem to be solved by the present application is how to quantitatively evaluate the output results of the multi-modal large model.

[0005] The technical solution adopted by the present application to solve the technical problem is:

[0006] In a first aspect, the present application provides a multi-modal large model quantitative evaluation method, comprising the following steps:

[0007] S1, obtaining a training set;

[0008] S2, constructing a model for quantitatively evaluating the output results of the multi-modal large model;

[0009] S3, training the model for quantitatively evaluating the output results of the multi-modal large model using the training set obtained in step S1;

[0010] S4, evaluating the output results of the multi-modal large model using the trained model for quantitatively evaluating the output results of the multi-modal large model.

[0011] Based on the first aspect, further, the text pair in the training set and the similarity value corresponding to the text pair can be obtained by the following method: inputting the picture and the prompt text corresponding to the picture into a first multi-modal large model, the multi-modal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; inputting the picture and the prompt text corresponding to the picture into a second multi-modal large model, the multi-modal large model outputs an eighth text corresponding to the picture and the prompt text corresponding to the picture; inputting the sixth text and the eighth text into a general large model, the general large model outputs a similarity value of the sixth text and the eighth text; the sixth text and the eighth text constitute a text pair, and the similarity value of the sixth text and the eighth text output by the general large model is taken as the similarity value corresponding to the text pair constituted by the sixth text and the eighth text in the training set.

[0012] Based on the first aspect, further, the model for quantitatively evaluating the output result of the multi-modal large model is constructed by taking a BERT model as a benchmark model, adding a similarity task head and a similarity function; inputting the text pair into the BERT model, adjusting the output of the BERT model by a first method and taking the adjusted output as the input of the similarity task head, taking the output of the similarity task head as the input of the similarity function, and taking the output of the similarity function as the output of the model for quantitatively evaluating the output result of the multi-modal large model.

[0013] Based on the first aspect, further, the inputting of the text pair into the model for quantitatively evaluating the output result of the multi-modal large model is that the BERT model in the model for quantitatively evaluating the output result of the multi-modal large model outputs a first feature matrix and a second feature matrix; the first feature matrix corresponds to one of the two texts in the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model, and the second feature matrix corresponds to the other of the two texts in the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model.

[0014] The first feature matrix is an M*N matrix:

[0015]

[0016] wherein M is the number of rows of the matrix, and N is the number of columns of the matrix; is an element in the first feature matrix.

[0017] The second feature matrix is an L*N matrix:

[0018]

[0019] wherein L is the number of rows of the matrix, and N is the number of columns of the matrix; is an element in the second feature matrix.

[0020] Based on the first aspect, further, the first method refers to taking the first row vector of the first feature matrix as a third feature vector corresponding to the first feature matrix, taking the first row vector of the second feature matrix as a fourth feature vector corresponding to the second feature matrix, and taking the third feature vector and the fourth feature vector as inputs of the similarity task head.

[0021] Based on the first aspect, further, the similarity task head is composed of a first linear layer, a first activation function, a second linear layer and an activation function; the input of the first linear layer is taken as the input of the similarity task head, the output of the first linear layer is taken as the input of the first activation function, the output of the first activation function is taken as the input of the second linear layer, and the output of the second linear layer is taken as the output of the similarity task head.

[0022] The formula of the first linear layer is as follows:

[0023] Linear1(x l1 )=W1x l1 +b1;

[0024] Wherein, Linear1(x l1 ) is the first linear layer; x l1 is the input of the first linear layer, W1 is the first network weight, and b1 is the first bias.

[0025] The formula of the second linear layer is as follows:

[0026] Linear2(x l2 )=W2x l2 +b2;

[0027] Wherein, Linear2(x l2 ) is the second linear layer; x l2 is the input of the second linear layer, W2 is the second network weight, and b2 is the second bias.

[0028] The formula of the first activation function is as follows:

[0029]

[0030] Wherein, GELU(x g1 ) is the first activation function; x g1 is the input of the first activation function, μ is the expectation, and δ is the standard deviation.

[0031] The third feature vector is input into the similarity task head, and the similarity task head outputs a fifth feature vector corresponding to the third feature vector. The fourth feature vector is input into the similarity task head, and the similarity task head outputs a sixth feature vector corresponding to the fourth feature vector.

[0032] Based on the first aspect, further, the formula of the similarity function is as follows:

[0033]

[0034] wherein f s is a similarity function, is the i-th element in the fifth feature vector; is the i-th element in the sixth feature vector.

[0035] Based on the first aspect, further, the model for quantitatively evaluating the output result of the multi-modal large model adopts the following loss function:

[0036] f l (y p ,y t )=(y p -y t ) 2 ;

[0037] wherein f l (y p ,y t ) is a loss function; y p is a similarity value output by the model for quantitatively evaluating the output result of the multi-modal large model corresponding to the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model; y t is a similarity value in the training set corresponding to the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model.

[0038] Based on the first aspect, further, the method for evaluating the output result of the multi-modal large model by using the trained model for quantitatively evaluating the output result of the multi-modal large model is as follows:

[0039] inputting a picture into the multi-modal large model to be evaluated to obtain a text output by the multi-modal large model to be evaluated; constructing a text pair from the real label text corresponding to the picture and the text output by the multi-modal large model to be evaluated; inputting the text pair into the trained model for quantitatively evaluating the output result of the multi-modal large model; and outputting, by the trained model for quantitatively evaluating the output result of the multi-modal large model, a similarity value corresponding to the text pair.

[0040] The second aspect of the present application provides a multi-modal large model quantitative evaluation system, characterized in that it comprises a data acquisition module, a model construction module, a model testing module and a result output module, wherein:

[0041] The data acquisition module is used to acquire a training set.

[0042] a model construction module, configured to construct a model for quantitatively evaluating output results of the multi-modal large model;

[0043] a model test module, configured to train the model for quantitatively evaluating output results of the multi-modal large model by using the training set obtained by the data acquisition module;

[0044] a result output module, configured to evaluate output results of the multi-modal large model by using the trained model for quantitatively evaluating output results of the multi-modal large model.

[0045] In a third aspect, the present application also provides an electronic device, including a memory, configured to store one or more programs; and a processor. When the one or more programs are executed by the processor, the method in any one of the above first aspect is implemented.

[0046] In a fourth aspect, the present application also provides a computer readable storage medium, having a computer program stored thereon. When the computer program is executed by a processor, the method in any one of the above first aspect is implemented.

[0047] The present application has the following beneficial effects:

[0048] The multi-modal large model quantitative evaluation method of the present application can be used to quantitatively evaluate output results of the multi-modal large model. BRIEF DESCRIPTION OF DRAWINGS

[0049] Figure 1 The flowchart of the multi-modal large model quantitative evaluation method provided by the present application is shown.

[0050] Figure 2 The principle block diagram of the multi-modal large model quantitative evaluation system provided by the present application is shown.

[0051] Figure 3 The structural block diagram of the electronic device provided by the present application is shown.

[0052] The reference signs are: 100, data acquisition module; 200, model construction module; 300, model test module; 400, result output module; 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical scheme and advantages of the embodiments of the present application clearer, the technical scheme of the embodiments of the present application will be described clearly and completely below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are some embodiments of the present application, not all embodiments of the present application. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the application provided in the accompanying drawings is not intended to limit the scope of the application claimed, but merely represents selected embodiments of the application. Based on the embodiments in the application, all other embodiments obtained by those of ordinary skill in the art without creative labor are within the scope of protection of the application.

[0055] It should be noted that similar reference numbers and letters represent similar items in the following drawings, so once an item is defined in one drawing, it does not need to be further defined and explained in subsequent drawings.

[0056] It should be noted that in this paper, relationship terms such as first and second are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between the entities or operations. Moreover, the terms "include", "contain" or any other variants thereof are intended to cover non-exclusive inclusion, so that the process, method, article or equipment including a series of elements not only includes those elements, but also includes other elements not explicitly listed or inherent to such process, method, article or equipment. Without more limitation, the element defined by the statement "including a" does not exclude the presence of other identical elements in the process, method, article or equipment including the element.

[0057] In the description of the embodiments of the application, "a plurality of" represents at least 2.

[0058] As shown in the accompanying drawings, one embodiment of the application provides a multi-modal large model quantitative evaluation method, which includes the following steps: Figure 1

[0059] S1, obtaining a training set;

[0060] S2, constructing a model for quantitatively evaluating the output results of the multi-modal large model;

[0061] S3, training the model for quantitatively evaluating the output results of the multi-modal large model using the training set obtained in step S1;

[0062] S4, evaluating the output results of the multi-modal large model using the trained model for quantitatively evaluating the output results of the multi-modal large model.

[0063] The training set includes a plurality of text pairs and similarity values corresponding to the text pairs.

[0064] The value range of the similarity value is [0, 1]. 0 represents complete dissimilarity, and 1 represents complete identity. The greater the similarity value, the higher the similarity of the text pair. ​

[0065] In an embodiment of the present application, the text pair in the training set and the similarity value corresponding to the text pair can be obtained by the following method: inputting the picture into the first multi-modal large model, the multi-modal large model outputs the first text corresponding to the picture; inputting the first text into the general large model, the general large model outputs the second text which has the same meaning but different expression as the first text; the text pair is composed of the first text and the second text, and the similarity value corresponding to the text pair in the training set is marked as 1.

[0066] In an embodiment of the present application, the general large model is chatgpt.

[0067] In an embodiment of the present application, the text pair in the training set and the similarity value corresponding to the text pair can also be obtained by the following method: inputting the picture into the first multi-modal large model, the multi-modal large model outputs the first text corresponding to the picture; inputting the picture into the second multi-modal large model, the multi-modal large model outputs the third text corresponding to the picture; inputting the first text and the third text into the general large model, the general large model outputs the similarity value of the first text and the third text; the text pair is composed of the first text and the third text, and the similarity value of the first text and the third text output by the general large model is taken as the similarity value corresponding to the text pair composed of the first text and the third text in the training set.

[0068] In an embodiment of the present application, the text pair in the training set and the similarity value corresponding to the text pair can also be obtained by the following method: inputting the picture into the first multi-modal large model, the multi-modal large model outputs the first text corresponding to the picture; inputting two different first texts into the general large model, the general large model outputs the similarity value of the two different first texts; the text pair is composed of the two different first texts, and the similarity value of the two different first texts output by the general large model is taken as the similarity value corresponding to the text pair composed of the two different first texts in the training set.

[0069] In an embodiment of the present application, the text pair in the training set and the similarity value corresponding to the text pair can also be obtained by the following method: for any fourth text, the fifth text which has the same meaning but different expression as the fourth text is given by the industry expert; the text pair is composed of the fourth text and the fifth text, and the similarity value corresponding to the text pair in the training set is marked as 1.

[0070] In an embodiment of the present application, the text pairs in the training set and the similarity values corresponding to the text pairs can also be obtained by the following method: for any two fourth texts, a similarity value between the two fourth texts is given by an industry expert; a text pair is formed by the two fourth texts, and the similarity value between the two fourth texts given by the industry expert is taken as the similarity value corresponding to the text pair formed by the two fourth texts in the training set.

[0071] In an embodiment of the present application, the text pairs in the training set and the similarity values corresponding to the text pairs can also be obtained by the following method: the picture and the prompt text corresponding to the picture are input into a first multi-modal large model, the multi-modal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; the sixth text is input into a general large model, and the general large model outputs a seventh text with the same meaning but different expression as the sixth text; a text pair is formed by the sixth text and the seventh text, and the similarity value corresponding to the text pair in the training set is marked as 1.

[0072] In an embodiment of the present application, the prompt text corresponding to the picture is "Please describe the possible urban management problems according to the following picture".

[0073] In an embodiment of the present application, the text pairs in the training set and the similarity values corresponding to the text pairs can also be obtained by the following method: the picture and the prompt text corresponding to the picture are input into a first multi-modal large model, the multi-modal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; the picture and the prompt text corresponding to the picture are input into a second multi-modal large model, the multi-modal large model outputs an eighth text corresponding to the picture and the prompt text corresponding to the picture; the sixth text and the eighth text are input into a general large model, and the general large model outputs a similarity value of the sixth text and the eighth text; a text pair is formed by the sixth text and the eighth text, and the similarity value of the sixth text and the eighth text output by the general large model is taken as the similarity value corresponding to the text pair formed by the sixth text and the eighth text in the training set.

[0074] In an embodiment of the present application, the text pairs in the training set and the similarity values corresponding to the text pairs can also be obtained by the following method: the picture and the prompt text corresponding to the picture are input into a first multi-modal large model, the multi-modal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; two different sixth texts are input into a general large model, and the general large model outputs a similarity value of the two different sixth texts; a text pair is formed by the two different sixth texts, and the similarity value of the two different sixth texts output by the general large model is taken as the similarity value corresponding to the text pair formed by the two different sixth texts in the training set.

[0075] In an embodiment of the present application, the model for quantitatively evaluating the output result of the multi-modal large model is constructed by taking a BERT model (Bidirectional Encoder Representations from Transformers) as a benchmark model, adding a similarity task head and a similarity function; inputting a text pair into the BERT model, adjusting the output of the BERT model by a first method to serve as the input of the similarity task head, taking the output of the similarity task head as the input of the similarity function, and taking the output of the similarity function as the output of the model for quantitatively evaluating the output result of the multi-modal large model.

[0076] The output of the model for quantitatively evaluating the output result of the multi-modal large model is a similarity value corresponding to the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model, which is given by the model for quantitatively evaluating the output result of the multi-modal large model.

[0077] In an embodiment of the present application, the text pair is input into the model for quantitatively evaluating the output result of the multi-modal large model, and the BERT model in the model for quantitatively evaluating the output result of the multi-modal large model outputs a first feature matrix and a second feature matrix; the first feature matrix corresponds to one of the texts in the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model, and the second feature matrix corresponds to the other text in the text pair input into the model for quantitatively evaluating the output result of the multi-modal large model.

[0078] The first feature matrix is an MxN matrix:

[0079]

[0080] wherein M is the number of rows of the matrix, and N is the number of columns of the matrix. is an element in the first feature matrix.

[0081] The second feature matrix is an LxN matrix:

[0082]

[0083] wherein L is the number of rows of the matrix, and N is the number of columns of the matrix. is an element in the second feature matrix.

[0084] In one embodiment of this application, the first method refers to using the first row vector of the first feature matrix as the third feature vector corresponding to the first feature matrix, using the first row vector of the second feature matrix as the fourth feature vector corresponding to the second feature matrix, and using the third feature vector and the fourth feature vector as input to the similarity task head.

[0085] The similarity task head consists of multiple linear layers and activation functions.

[0086] In one embodiment of this application, the similarity task head consists of two linear layers and an activation function. In this embodiment, the similarity task head includes a first linear layer, a first activation function, and a second linear layer. The input of the first linear layer serves as the input of the similarity task head, the output of the first linear layer serves as the input of the first activation function, the output of the first activation function serves as the input of the second linear layer, and the output of the second linear layer serves as the output of the similarity task head.

[0087] The formula for the first linear layer is as follows:

[0088] Linear1(x l1 )=W1x l1 +b1;

[0089] Among them, Linear1(x l1 ) is the first linear layer; x l1 W1 is the input of the first linear layer, W1 is the first network weight, and b1 is the first bias.

[0090] The formula for the second linear layer is as follows:

[0091] Linear2(x l2 ) = W2x l2 +b2;

[0092] Among them, Linear2(x l2 ) is the second linear layer; x l2 W1 is the input to the second linear layer, W2 is the second network weight, and b2 is the second bias.

[0093] The formula for the first activation function is as follows:

[0094]

[0095] Among them, GELU(x g1 ) is the first activation function; x g1 δ is the input to the first activation function, μ is the expectation, and δ is the standard deviation.

[0096] The third feature vector is input into the similarity task head, and the similarity task head outputs a fifth feature vector corresponding to the third feature vector. The fourth feature vector is input into the similarity task head, and the similarity task head outputs a sixth feature vector corresponding to the fourth feature vector.

[0097] In an embodiment of the present application, the formula of the similarity function is as follows:

[0098]

[0099] wherein f s is a similarity function, V i c is the i-th element in the fifth feature vector; V i d is the i-th element in the sixth feature vector.

[0100] In an embodiment of the present application, the model for quantitatively evaluating the output results of the multi-modal large model adopts the following loss function:

[0101] f l (y p ,y t )=(y p -y t ) 2 ;

[0102] wherein f l (y p ,y t ) is a loss function; y p is a similarity value output by the model for quantitatively evaluating the output results of the multi-modal large model corresponding to a text pair input into the model for quantitatively evaluating the output results of the multi-modal large model; and y t is a similarity value in the training set corresponding to the text pair input into the model for quantitatively evaluating the output results of the multi-modal large model.

[0103] In an embodiment of the present application, the output results of the multi-modal large model can be evaluated by using the trained model for quantitatively evaluating the output results of the multi-modal large model by using the following method:

[0104] The picture is input into the multi-modal large model to be evaluated to obtain a text output by the multi-modal large model to be evaluated. A text pair is formed by a real label text corresponding to the picture and the text output by the multi-modal large model to be evaluated. The text pair is input into the trained model for quantitatively evaluating the output results of the multi-modal large model, and the trained model for quantitatively evaluating the output results of the multi-modal large model outputs a similarity value corresponding to the text pair. The similarity value is the quantitative evaluation result given by the multi-modal large model quantitative evaluation method of the present application.

[0105] The real label text corresponding to the picture can be given by an industry expert.

[0106] In an embodiment of the present application, the model trained for quantitatively evaluating the output result of the multi-modal large model is used to evaluate the output result of the multi-modal large model, which can also adopt the following method:

[0107] The picture and the prompt text corresponding to the picture are input into the multi-modal large model to be evaluated to obtain the text output by the multi-modal large model to be evaluated; the real label text corresponding to the picture and the prompt text is paired with the text output by the multi-modal large model to be evaluated to form a text pair, and the text pair is input into the model trained for quantitatively evaluating the output result of the multi-modal large model, so that the model trained for quantitatively evaluating the output result of the multi-modal large model outputs a similarity value corresponding to the text pair. The similarity value is the quantitative evaluation result given by the multi-modal large model quantitative evaluation method of the present application.

[0108] The real label text corresponding to the picture and the prompt text can be given by an industry expert.

[0109] As shown in Figure 2 An embodiment of the present application provides a multi-modal large model quantitative evaluation system, which includes a data acquisition module 100, a model construction module 200, a model test module 300, and a result output module 400, wherein:

[0110] The data acquisition module 100 is configured to obtain a training set.

[0111] The model construction module 200 is configured to construct a model for quantitatively evaluating the output result of the multi-modal large model.

[0112] The model test module 300 is configured to train the model for quantitatively evaluating the output result of the multi-modal large model by using the training set obtained by the data acquisition module 100.

[0113] The result output module 400 is configured to evaluate the output result of the multi-modal large model by using the model trained for quantitatively evaluating the output result of the multi-modal large model.

[0114] As shown in Figure 3 In a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory 101 configured to store one or more programs, and a processor 102. When the one or more programs are executed by the processor 102, the method of any one of the above-mentioned first aspect is implemented.

[0115] The communication interface 103 is also included, and the memory 101, the processor 102 and the communication interface 103 are electrically connected to each other directly or indirectly to realize the transmission or interaction of data. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 can execute various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used for signaling or data communication with other node devices.

[0116] The memory 101 can be, but is not limited to, a random access memory (RAM), a read-only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable programmable read-only memory (EEPROM) and the like.

[0117] The processor 102 can be an integrated circuit chip with signal processing capability. The processor 102 can be a general-purpose processor, including a central processing unit (CPU), a network processor (NP) and the like; it can also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components.

[0118] In a fourth aspect, the embodiments of the present application further provide a computer readable storage medium, which stores a computer program. The computer program is executed by the processor 102 to implement the method of any one of the above first aspect. The functions can be implemented in the form of software function modules and sold or used as independent products. Based on this understanding, the technical solutions of the present application can be embodied in the form of a software product and stored in a storage medium, including a number of instructions to make a computer device (which can be a personal computer, a server, or a network device, etc.) execute all or part of the steps of the methods described in the embodiments of the present application. The foregoing storage medium includes: a U disk, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a magnetic disk or an optical disk, and various media that can store program codes.

[0119] The above only is the preferred embodiment of the present application, and is not used to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principles of the present application shall be included in the protection scope of the present application.

[0120] It is obvious for those skilled in the art that the present application is not limited to the details of the above exemplary embodiments, and the present application can be implemented in other specific forms without departing from the spirit or essential characteristics of the present application. Therefore, the embodiments should be regarded as exemplary and non-limiting, and the scope of the present application is defined by the appended claims rather than the above description, and all changes falling within the meaning and scope of the equivalent elements of the claims are intended to be included in the present application. Any reference signs in the claims should not be regarded as limiting the claims.

Claims

1. A method for quantitative evaluation of multimodal large models, characterized in that, Includes the following steps: S1, Obtain the training set; S2, Construct a model for quantitatively evaluating the output results of a large multimodal model; S3, Use the training set obtained in step S1 to train a model for quantitatively evaluating the output results of a multimodal large model; S4. Use the trained model to evaluate the output of the multimodal large model. The text pairs in the training set and their corresponding similarity values ​​are obtained using the following method: An image and its corresponding prompt text are input into a first multimodal large model, which outputs a sixth text corresponding to the image and its corresponding prompt text; the image and its corresponding prompt text are input into a second multimodal large model, which outputs an eighth text corresponding to the image and its corresponding prompt text; the sixth and eighth texts are input into a general large model, which outputs a similarity value between the sixth and eighth texts; a text pair is formed from the sixth and eighth texts, and the similarity value between the sixth and eighth texts output by the general large model is used as the similarity value corresponding to the text pair formed by the sixth and eighth texts in the training set; The model used to quantify the output of a multimodal large model is constructed by adding a similarity task head and a similarity function to the BERT model as the baseline model. Text pairs are input to the model, and the BERT model in the model outputs a first feature matrix and a second feature matrix. The first and second feature matrices are adjusted using a first method to obtain a third and a fourth feature vector. The third and fourth feature vectors are used as input to the similarity task head, which outputs a fifth and a sixth feature vector. The fifth and sixth feature vectors are used as input to the similarity function, and the value of the similarity function is calculated. The value of the similarity function is then used as the output of the model used to quantify the output of the multimodal large model. The first method refers to taking the first row vector of the first feature matrix as the third feature vector corresponding to the first feature matrix, and taking the first row vector of the second feature matrix as the fourth feature vector corresponding to the second feature matrix. The formula for the similarity function is as follows: ； in, It is a similarity function. It is the fifth eigenvector. One element; It is the sixth eigenvector. Each element.

2. The multimodal large model quantitative evaluation method according to claim 1, characterized in that, The first feature matrix corresponds to one of the text pairs of the model used to quantify the output of the multimodal large model, and the second feature matrix corresponds to the other text pair of the model used to quantify the output of the multimodal large model.

3. The multimodal large model quantitative evaluation method according to claim 1, characterized in that, The similarity task head consists of a first linear layer, a first activation function, a second linear layer, and an activation function; the input of the first linear layer is used as the input of the similarity task head, the output of the first linear layer is used as the input of the first activation function, the output of the first activation function is used as the input of the second linear layer, and the output of the second linear layer is used as the output of the similarity task head. The formula for the first linear layer is as follows: ； in, It is the first linear layer; It is the input of the first linear layer. It is the first network weight. This is the first deviation; The formula for the second linear layer is as follows: ； in, It is the second linear layer; It is the input of the second linear layer. It is the second network weight. This is the second deviation; The formula for the first activation function is as follows: ； in, It is the first activation function; It is the input to the first activation function. It is an expectation. That is the standard deviation.

4. The multimodal large model quantitative evaluation method according to claim 1, characterized in that, The model used to quantitatively evaluate the output of the multimodal large model employs the following loss function: ； in, It is a loss function; It is the similarity value between the text pairs of the input model used to quantify the output of the multimodal large model and the output of the corresponding model used to quantify the output of the multimodal large model; It is the similarity value in the training set corresponding to the text pairs of the model used to quantify the output results of the multimodal large model.

5. The multimodal large model quantitative evaluation method according to claim 1, characterized in that, The method for evaluating the output of a multimodal large model using the trained model to quantitatively evaluate the output of the multimodal large model is as follows: The image and its corresponding prompt text are input into the multimodal large model to be evaluated, and the output text of the multimodal large model is obtained. The ground truth label text corresponding to the image and its prompt text and the output text of the multimodal large model are used to form a text pair. The text pair is input into the trained model used to quantify the output results of the multimodal large model. The trained model used to quantify the output results of the multimodal large model outputs the similarity value corresponding to the text pair.

6. A multimodal large-scale model quantitative evaluation system, characterized in that, It includes a data acquisition module, a model building module, a model testing module, and a results output module, among which: The data acquisition module is used to acquire the training set; The model building module is used to build models for quantitatively evaluating the output results of large multimodal models; The model testing module is used to train a model for quantitatively evaluating the output results of a large multimodal model using the training set obtained from the data acquisition module. The results output module is used to evaluate the output of the multimodal large model using the trained model for quantitative evaluation of the output results of the multimodal large model. The text pairs in the training set and their corresponding similarity values ​​are obtained using the following method: An image and its corresponding prompt text are input into a first multimodal large model, which outputs a sixth text corresponding to the image and its corresponding prompt text; the image and its corresponding prompt text are input into a second multimodal large model, which outputs an eighth text corresponding to the image and its corresponding prompt text; the sixth and eighth texts are input into a general large model, which outputs a similarity value between the sixth and eighth texts; a text pair is formed from the sixth and eighth texts, and the similarity value between the sixth and eighth texts output by the general large model is used as the similarity value corresponding to the text pair formed by the sixth and eighth texts in the training set; The model used to quantify the output of a multimodal large model is constructed by adding a similarity task head and a similarity function to the BERT model as the baseline model. Text pairs are input to the model, and the BERT model in the model outputs a first feature matrix and a second feature matrix. The first and second feature matrices are adjusted using a first method to obtain a third and a fourth feature vector. The third and fourth feature vectors are used as input to the similarity task head, which outputs a fifth and a sixth feature vector. The fifth and sixth feature vectors are used as input to the similarity function, and the value of the similarity function is calculated. The value of the similarity function is then used as the output of the model used to quantify the output of the multimodal large model. The first method refers to taking the first row vector of the first feature matrix as the third feature vector corresponding to the first feature matrix, and taking the first row vector of the second feature matrix as the fourth feature vector corresponding to the second feature matrix. The formula for the similarity function is as follows: ； in, It is a similarity function. It is the fifth eigenvector. One element; It is the sixth eigenvector. Each element.

Images