Multi-modal large model quantitative evaluation method and system

Abstract

The invention discloses a multi-modal large model quantitative evaluation method and system, and relates to the technical field of artificial intelligence. The method comprises the following steps: S1, acquiring a training set; s2, constructing a model for quantitatively evaluating an output result of the multi-modal large model; s3, training a model used for quantitatively evaluating an output result of the multi-modal large model by using the training set obtained in the step S1; and S4, evaluating the output result of the multi-modal large model by using the trained model for quantitatively evaluating the output result of the multi-modal large model. The invention discloses a multi-modal large model quantitative evaluation method and system, and is used for carrying out quantitative evaluation on an output result of a multi-modal large model.

Description

Technical Field

[0001] The present application relates to the field of artificial intelligence technology, and in particular to a multi-modal large model quantitative evaluation method and system. Background Art

[0002] The breakthrough of multimodal large models has brought about the ability to solve various complex tasks of images and texts, and many models are close to or even reach the performance level of human experts. Multimodal large models aim to achieve more powerful general intelligence by combining LLMs (i.e. large language models) with multimodal inputs, and solve various complex multimodal tasks, such as visual recognition ability, knowledge ability, character recognition ability, mathematical ability, scene text understanding, etc. However, there is no unified method to evaluate the model capabilities of the complex and diverse text output content of the current multimodal large models, and there is no quantitative evaluation method for smart city multimodal large models, which makes it impossible to effectively quantify the capabilities of trained multimodal large models.

[0003] The existing evaluation method is to make the same character judgment for each position of the content predicted by the multimodal large model and the corresponding label position. The model prediction results using this method must ensure the assumption that the model output is consistent with the label, but it cannot achieve accurate judgment of texts with the same meaning but different expressions; some people also use other large models to evaluate the text predicted by the multimodal large model and the corresponding label text to obtain an evaluation value. This method uses the powerful capabilities of other large models for evaluation. Although it can partially achieve the judgment of content with the same meaning but different expressions, it still has unreliable judgments, especially text evaluation in areas where other large models are not good at. Summary of the invention

[0004] The technical problem to be solved by this application is: how to quantitatively evaluate the output results of a large multimodal model.

[0005] The technical solution adopted by this application to solve its technical problem is:

[0006] In a first aspect, the present application provides a multimodal large model quantitative evaluation method, comprising the following steps:

[0007] S1, obtain the training set;

[0008] S2, building a model for quantitatively evaluating the output results of multimodal large models;

[0009] S3, using the training set obtained in step S1 to train a model for quantitatively evaluating the output results of the multimodal large model;

[0010] S4, evaluating the output results of the multimodal large model using the trained model for quantitatively evaluating the output results of the multimodal large model.

[0011] Based on the first aspect, further, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can be obtained by the following method: input the picture and the prompt text corresponding to the picture into the first multimodal large model, and the multimodal large model outputs the sixth text corresponding to the picture and the prompt text corresponding to the picture; input the picture and the prompt text corresponding to the picture into the second multimodal large model, and the multimodal large model outputs the eighth text corresponding to the picture and the prompt text corresponding to the picture; input the sixth text and the eighth text into the general large model, and the general large model outputs the similarity value of the sixth text and the eighth text; the sixth text and the eighth text constitute a text pair, and the similarity value of the sixth text and the eighth text output by the general large model is used as the similarity value corresponding to the text pair constituted by the sixth text and the eighth text in the training set.

[0012] Based on the first aspect, further, the above-mentioned model for quantitatively evaluating the output results of the multimodal large model is constructed by taking the BERT model as the baseline model and adding a similarity task head and a similarity function; the text pair is input into the BERT model, the output of the BERT model is adjusted using the first method as the input of the similarity task head, the output of the similarity task head is used as the input of the similarity function, and the output of the similarity function is used as the output of the model for quantitatively evaluating the output results of the multimodal large model.

[0013] Based on the first aspect, further, the above-mentioned text pair is input into the model for quantitatively evaluating the output results of the multimodal large model, and the BERT model in the model for quantitatively evaluating the output results of the multimodal large model outputs a first feature matrix and a second feature matrix; the first feature matrix corresponds to one of the texts in the text pair input into the model for quantitatively evaluating the output results of the multimodal large model, and the second feature matrix corresponds to the other text in the text pair input into the model for quantitatively evaluating the output results of the multimodal large model.

[0014] The first characteristic matrix is ​​an M×N matrix:

[0015]

[0016] Where M is the number of rows of the matrix, and N is the number of columns of the matrix; is the element in the first characteristic matrix.

[0017] The second characteristic matrix is ​​an L×N matrix:

[0018]

[0019] Where L is the number of rows of the matrix, and N is the number of columns of the matrix; is an element in the second characteristic matrix.

[0020] Based on the first aspect, further, the above-mentioned first method refers to using the first row vector of the first feature matrix as the third eigenvector corresponding to the first feature matrix, using the first row vector of the second feature matrix as the fourth eigenvector corresponding to the second feature matrix, and using the third eigenvector and the fourth eigenvector as inputs of the similarity task head.

[0021] Based on the first aspect, further, the similarity task head is composed of a first linear layer, a first activation function, a second linear layer and an activation function; the input of the first linear layer is used as the input of the similarity task head, the output of the first linear layer is used as the input of the first activation function, the output of the first activation function is used as the input of the second linear layer, and the output of the second linear layer is used as the output of the similarity task head.

[0022] The formula for the first linear layer is as follows:

[0023] Linear 1 (x l1 )=W 1 x l1 +b 1 ;

[0024] Among them, Linear 1 (x l1 ) is the first linear layer; x l1 is the input of the first linear layer, W 1 is the first network weight, b 1 It is the first deviation.

[0025] The formula for the second linear layer is as follows:

[0026] Linear 2 (x l2 )=W 2 x l2 +b 2 ;

[0027] Among them, Linear 2 (x l2 ) is the second linear layer; x l2 is the input of the second linear layer, W 2 is the second network weight, b 2 It is the second deviation.

[0028] The formula of the first activation function is as follows:

[0029]

[0030] Among them, GELU(x g1 ) is the first activation function; x g1 is the input of the first activation function, μ is the expectation, and δ is the standard deviation.

[0031] The third feature vector is input into the similarity task head, and the similarity task head outputs a fifth feature vector corresponding to the third feature vector. The fourth feature vector is input into the similarity task head, and the similarity task head outputs a sixth feature vector corresponding to the fourth feature vector.

[0032] Based on the first aspect, further, the formula of the similarity function is as follows:

[0033]

[0034] Among them, f s is the similarity function, is the i-th element in the fifth eigenvector; is the i-th element in the sixth eigenvector.

[0035] Based on the first aspect, further, the above-mentioned model for quantitatively evaluating the output results of the multimodal large model adopts the following loss function:

[0036] f l (y p ,y t )=(y p -y t ) 2 ;

[0037] Among them, f l (y p ,y t ) is the loss function; y p is the similarity value of the model output for quantitatively evaluating the output result of the multimodal large model corresponding to the text pair input into the model for quantitatively evaluating the output result of the multimodal large model; y t It is the similarity value in the training set corresponding to the text pair input to the model for quantitatively evaluating the output results of the multimodal large model.

[0038] Based on the first aspect, further, the output result of the multimodal large model is evaluated by using the model for quantitatively evaluating the output result of the multimodal large model after training, using the following method:

[0039] An image is input into the multimodal large model to be evaluated to obtain the text output by the multimodal large model to be evaluated; a text pair is formed by the real label text corresponding to the image and the text output by the multimodal large model to be evaluated, and the text pair is input into a trained model for quantitatively evaluating the output results of the multimodal large model; the trained model for quantitatively evaluating the output results of the multimodal large model outputs a similarity value corresponding to the text pair.

[0040] In a second aspect, the present application provides a multimodal large model quantitative evaluation system, characterized in that it includes a data acquisition module, a model construction module, a model testing module and a result output module, wherein:

[0041] Data acquisition module, used to obtain training sets;

[0042] A model building module, used to build a model for quantitatively evaluating the output results of a multimodal large model;

[0043] A model testing module, used for training a model for quantitatively evaluating the output results of a multimodal large model using a training set obtained by the data acquisition module;

[0044] The result output module is used to evaluate the output results of the multimodal large model using the trained model for quantitatively evaluating the output results of the multimodal large model.

[0045] In a third aspect, the present application further provides an electronic device, comprising a memory for storing one or more programs and a processor. When the one or more programs are executed by the processor, any method in the first aspect described above is implemented.

[0046] In a fourth aspect, the present application further provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements any one of the methods in the first aspect described above.

[0047] The beneficial effects of this application are:

[0048] The multimodal large model quantitative evaluation method of the present application can be used to perform quantitative evaluation on the output results of the multimodal large model. BRIEF DESCRIPTION OF THE DRAWINGS

[0049] Figure 1 A flowchart of a multimodal large model quantitative evaluation method provided in an embodiment of the present application.

[0050] Figure 2 This is a principle block diagram of the multimodal large model quantitative evaluation system provided in an embodiment of the present application.

[0051] Figure 3 A structural block diagram of an electronic device provided in an embodiment of the present application.

[0052] The reference numerals are: 100, data acquisition module; 200, model building module; 300, model testing module; 400, result output module; 101, memory; 102, processor; 103, communication interface. DETAILED DESCRIPTION

[0053] In order to make the purpose, technical solution and advantages of the embodiments of the present application clearer, the technical solution in the embodiments of the present application will be clearly and completely described below in conjunction with the drawings in the embodiments of the present application. Obviously, the described embodiments are part of the embodiments of the present application, rather than all the embodiments. The components of the embodiments of the present application described and shown in the drawings here can be arranged and designed in various different configurations.

[0054] Therefore, the following detailed description of the embodiments of the present application provided in the accompanying drawings is not intended to limit the scope of the present application for which protection is sought, but merely represents selected embodiments of the present application. Based on the embodiments in the present application, all other embodiments obtained by ordinary technicians in the field without creative work are within the scope of protection of the present application.

[0055] It should be noted that similar reference numerals and letters denote similar items in the following drawings, and therefore, once an item is defined in one drawing, further definition and explanation thereof is not required in subsequent drawings.

[0056] It should be noted that, in this article, relational terms such as first and second, etc. are only used to distinguish one entity or operation from another entity or operation, and do not necessarily require or imply any such actual relationship or order between these entities or operations. Moreover, the terms "include", "comprise" or any other variants thereof are intended to cover non-exclusive inclusion, so that a process, method, article or device including a series of elements includes not only those elements, but also other elements not explicitly listed, or also includes elements inherent to such process, method, article or device. In the absence of further restrictions, the elements defined by the sentence "comprise a ..." do not exclude the existence of other identical elements in the process, method, article or device including the elements.

[0057] In the description of the embodiments of the present application, "plurality" means at least 2.

[0058] like Figure 1 As shown, an embodiment of the present application provides a multi-modal large model quantitative evaluation method comprising the following steps:

[0059] S1, obtain the training set;

[0060] S2, building a model for quantitatively evaluating the output results of multimodal large models;

[0061] S3, using the training set obtained in step S1 to train a model for quantitatively evaluating the output results of the multimodal large model;

[0062] S4, evaluating the output results of the multimodal large model using the trained model for quantitatively evaluating the output results of the multimodal large model.

[0063] The training set includes a plurality of text pairs and similarity values ​​corresponding to the text pairs.

[0064] The similarity value ranges from [0,1]. 0 means completely dissimilar, and 1 means completely identical. The larger the similarity value, the higher the similarity between the text pairs.

[0065] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can be obtained by the following method: inputting an image into a first multimodal large model, and the multimodal large model outputs a first text corresponding to the image; inputting the first text into a universal large model, and the universal large model outputs a second text having the same meaning as the first text but a different expression; forming a text pair from the first text and the second text, and marking the similarity value corresponding to the text pair in the training set as 1.

[0066] In one embodiment of the present application, the generic large model is chatgpt.

[0067] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: inputting an image into a first multimodal large model, and the multimodal large model outputs a first text corresponding to the image; inputting the image into a second multimodal large model, and the multimodal large model outputs a third text corresponding to the image; inputting the first text and the third text into a universal large model, and the universal large model outputs a similarity value between the first text and the third text; forming a text pair from the first text and the third text, and using the similarity value between the first text and the third text output by the universal large model as the similarity value corresponding to the text pair formed by the first text and the third text in the training set.

[0068] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: inputting an image into a first multimodal large model, and the multimodal large model outputs a first text corresponding to the image; inputting two different first texts into a universal large model, and the universal large model outputs the similarity values ​​of the two different first texts; a text pair is formed by the two different first texts, and the similarity values ​​of the two different first texts output by the universal large model are used as the similarity values ​​corresponding to the text pair formed by the two different first texts in the training set.

[0069] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: for any fourth text, an industry expert provides a fifth text with the same meaning as the fourth text but a different expression; the fourth text and the fifth text constitute a text pair, and the similarity value corresponding to the text pair in the training set is marked as 1.

[0070] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: for any two fourth texts, industry experts give similarity values ​​to the two fourth texts; the two fourth texts constitute a text pair, and the similarity values ​​of the two fourth texts given by the industry experts are used as the similarity values ​​corresponding to the text pair constituted by the two fourth texts in the training set.

[0071] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: input the picture and the prompt text corresponding to the picture into the first multimodal large model, and the multimodal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; input the sixth text into the general large model, and the general large model outputs a seventh text with the same meaning as the sixth text but a different expression; the sixth text and the seventh text constitute a text pair, and the similarity value corresponding to the text pair in the training set is marked as 1.

[0072] In one embodiment of the present application, the prompt text corresponding to the picture is "Please describe possible urban management problems based on the picture below."

[0073] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: input the picture and the prompt text corresponding to the picture into the first multimodal large model, and the multimodal large model outputs the sixth text corresponding to the picture and the prompt text corresponding to the picture; input the picture and the prompt text corresponding to the picture into the second multimodal large model, and the multimodal large model outputs the eighth text corresponding to the picture and the prompt text corresponding to the picture; input the sixth text and the eighth text into the general large model, and the general large model outputs the similarity value of the sixth text and the eighth text; the sixth text and the eighth text constitute a text pair, and the similarity value of the sixth text and the eighth text output by the general large model is used as the similarity value corresponding to the text pair constituted by the sixth text and the eighth text in the training set.

[0074] In one embodiment of the present application, the text pairs in the training set and the similarity values ​​corresponding to the text pairs can also be obtained by the following method: input the image and the prompt text corresponding to the image into the first multimodal large model, and the multimodal large model outputs the sixth text corresponding to the image and the prompt text corresponding to the image; input two different sixth texts into the general large model, and the general large model outputs the similarity values ​​of the two different sixth texts; the two different sixth texts constitute a text pair, and the similarity values ​​of the two different sixth texts output by the general large model are used as the similarity values ​​corresponding to the text pair constituted by the two different sixth texts in the training set.

[0075] In one embodiment of the present application, the model for quantitatively evaluating the output results of the multimodal large model is constructed by taking the BERT model (Bidirectional Encoder Representations from Transformers, i.e., bidirectional encoder representations from Transformer; Transformer is a model) as a benchmark model, and adding a similarity task head and a similarity function; the text pair is input into the BERT model, the output of the BERT model is adjusted using the first method as the input of the similarity task head, the output of the similarity task head is used as the input of the similarity function, and the output of the similarity function is used as the output of the model for quantitatively evaluating the output results of the multimodal large model.

[0076] The output of the model for quantitatively evaluating the output results of the multimodal large model is the similarity value corresponding to the text pair input into the model for quantitatively evaluating the output results of the multimodal large model, and the similarity value is given by the model for quantitatively evaluating the output results of the multimodal large model.

[0077] In one embodiment of the present application, a text pair is input into a model for quantitatively evaluating an output result of a multimodal large model, and a BERT model in the model for quantitatively evaluating an output result of the multimodal large model outputs a first feature matrix and a second feature matrix; the first feature matrix corresponds to one of the texts in the text pair input into the model for quantitatively evaluating an output result of the multimodal large model, and the second feature matrix corresponds to the other text in the text pair input into the model for quantitatively evaluating an output result of the multimodal large model.

[0078] The first characteristic matrix is ​​an M×N matrix:

[0079]

[0080] Where M is the number of rows of the matrix, and N is the number of columns of the matrix; is the element in the first characteristic matrix.

[0081] The second characteristic matrix is ​​an L×N matrix:

[0082]

[0083] Where L is the number of rows of the matrix, and N is the number of columns of the matrix; is an element in the second characteristic matrix.

[0084] In one embodiment of the present application, the first method refers to using the first row vector of the first feature matrix as the third eigenvector corresponding to the first feature matrix, using the first row vector of the second feature matrix as the fourth eigenvector corresponding to the second feature matrix, and using the third eigenvector and the fourth eigenvector as inputs of the similarity task head.

[0085] The similarity task head consists of multiple linear layers and activation functions.

[0086] In one embodiment of the present application, the similarity task head is composed of two linear layers and an activation function. In this embodiment, the similarity task head includes a first linear layer, a first activation function, and a second linear layer. The input of the first linear layer is used as the input of the similarity task head, the output of the first linear layer is used as the input of the first activation function, the output of the first activation function is used as the input of the second linear layer, and the output of the second linear layer is used as the output of the similarity task head.

[0087] The formula for the first linear layer is as follows:

[0088] Linear 1 (x l1 )=W 1 x l1 +b 1 ;

[0089] Among them, Linear 1 (x l1 ) is the first linear layer; x l1 is the input of the first linear layer, W 1 is the first network weight, b 1 It is the first deviation.

[0090] The formula for the second linear layer is as follows:

[0091] Linear 2 (x l2 )=W 2 x l2 +b 2 ;

[0092] Among them, Linear 2 (x l2 ) is the second linear layer; x l2 is the input of the second linear layer, W2 is the second network weight, b 2 It is the second deviation.

[0093] The formula of the first activation function is as follows:

[0094]

[0095] Among them, GELU(x g1 ) is the first activation function; x g1 is the input of the first activation function, μ is the expectation, and δ is the standard deviation.

[0096] The third feature vector is input into the similarity task head, and the similarity task head outputs a fifth feature vector corresponding to the third feature vector. The fourth feature vector is input into the similarity task head, and the similarity task head outputs a sixth feature vector corresponding to the fourth feature vector.

[0097] In one embodiment of the present application, the formula of the similarity function is as follows:

[0098]

[0099] Among them, f s is the similarity function, V i c is the i-th element in the fifth eigenvector; V i d is the i-th element in the sixth eigenvector.

[0100] In one embodiment of the present application, the model used to quantitatively evaluate the output results of the multimodal large model adopts the following loss function:

[0101] f l (y p ,y t )=(y p -y t ) 2 ;

[0102] Among them, f l (y p ,y t ) is the loss function; y p is the similarity value of the model output for quantitatively evaluating the output result of the multimodal large model corresponding to the text pair input into the model for quantitatively evaluating the output result of the multimodal large model; y t It is the similarity value in the training set corresponding to the text pair input to the model for quantitatively evaluating the output results of the multimodal large model.

[0103] In one embodiment of the present application, the following method may be used to evaluate the output result of the multimodal large model using the trained model for quantitatively evaluating the output result of the multimodal large model:

[0104] Input the image into the multimodal large model to be evaluated, and obtain the text output by the multimodal large model to be evaluated; the real label text corresponding to the image and the text output by the multimodal large model to be evaluated constitute a text pair, and input the text pair into the trained model for quantitatively evaluating the output results of the multimodal large model, and the trained model for quantitatively evaluating the output results of the multimodal large model outputs the similarity value corresponding to the text pair. The similarity value is the quantitative evaluation result given by the multimodal large model quantitative evaluation method of this application.

[0105] The actual label text corresponding to the image can be given by industry experts.

[0106] In one embodiment of the present application, the following method may be used to evaluate the output result of the multimodal large model using the trained model for quantitatively evaluating the output result of the multimodal large model:

[0107] Input the image and the prompt text corresponding to the image into the multimodal large model to be evaluated, and obtain the text output by the multimodal large model to be evaluated; the real label text corresponding to the image and the prompt text corresponding to the image and the text output by the multimodal large model to be evaluated constitute a text pair, and input the text pair into the trained model for quantitatively evaluating the output results of the multimodal large model, and the trained model for quantitatively evaluating the output results of the multimodal large model outputs the similarity value corresponding to the text pair. The similarity value is the quantitative evaluation result given by the multimodal large model quantitative evaluation method of this application.

[0108] The real label text corresponding to the picture and the prompt text corresponding to the picture can be given by industry experts.

[0109] like Figure 2 As shown, a multimodal large model quantitative evaluation system provided by an embodiment of the present application includes a data acquisition module 100, a model construction module 200, a model testing module 300 and a result output module 400, wherein:

[0110] The data acquisition module 100 is used to obtain a training set;

[0111] A model building module 200, for building a model for quantitatively evaluating the output results of the multimodal large model;

[0112] A model testing module 300 is used to train a model for quantitatively evaluating the output results of the multimodal large model using the training set obtained by the data acquisition module 100;

[0113] The result output module 400 is used to evaluate the output result of the multimodal large model using the trained model for quantitatively evaluating the output result of the multimodal large model.

[0114] like Figure 3 As shown, in a third aspect, an embodiment of the present application further provides an electronic device, which includes a memory 101 for storing one or more programs and a processor 102. When the one or more programs are executed by the processor 102, any method in the first aspect described above is implemented.

[0115] The memory 101, the processor 102 and the communication interface 103 are directly or indirectly electrically connected to each other to achieve data transmission or interaction. For example, these elements can be electrically connected to each other through one or more communication buses or signal lines. The memory 101 can be used to store software programs and modules, and the processor 102 executes various functional applications and data processing by executing the software programs and modules stored in the memory 101. The communication interface 103 can be used to communicate signaling or data with other node devices.

[0116] Among them, the memory 101 can be, but is not limited to, a random access memory (RAM), a read only memory (ROM), a programmable read-only memory (PROM), an erasable programmable read-only memory (EPROM), an electrically erasable read-only memory (EEPROM), etc.

[0117] The processor 102 may be an integrated circuit chip with signal processing capability. The processor 102 may be a general-purpose processor, including a central processing unit (CPU), a network processor (NP), etc.; it may also be a digital signal processor (DSP), an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), or other programmable logic devices, discrete gate or transistor logic devices, or discrete hardware components.

[0118] In a fourth aspect, the embodiment of the present application further provides a computer-readable storage medium on which a computer program is stored, and when the computer program is executed by the processor 102, a method as in any one of the first aspects described above is implemented. If the function is implemented in the form of a software function module and sold or used as an independent product, it can be stored in a computer-readable storage medium. Based on such an understanding, the technical solution of the present application is essentially or the part that contributes to the prior art or the part of the technical solution can be embodied in the form of a software product, and the computer software product is stored in a storage medium, including several instructions for a computer device (which can be a personal computer, a server, or a network device, etc.) to perform all or part of the steps of the method described in each embodiment of the present application. The aforementioned storage medium includes: various media that can store program codes, such as a USB flash drive, a mobile hard disk, a read-only memory (ROM, Read-Only Memory), a random access memory (RAM, Random Access Memory), a disk or an optical disk.

[0119] The above are only preferred embodiments of the present application and are not intended to limit the present application. For those skilled in the art, the present application may have various modifications and variations. Any modification, equivalent replacement, improvement, etc. made within the spirit and principle of the present application shall be included in the protection scope of the present application.

[0120] It will be apparent to those skilled in the art that the present application is not limited to the details of the exemplary embodiments described above, and that the present application can be implemented in other specific forms without departing from the spirit or essential features of the present application. Therefore, the embodiments should be considered exemplary and non-limiting in all respects, and the scope of the present application is defined by the appended claims rather than the above description, and it is intended that all changes falling within the meaning and scope of the equivalent elements of the claims be included in the present application. Any reference numeral in a claim should not be considered as limiting the claim to which it relates.

Claims

1. A multi-modal large model quantitative evaluation method, characterized in that: The following steps are involved: S1, obtain the training set; S2, building a model for quantitatively evaluating the output results of multimodal large models; S3, using the training set obtained in step S1 to train a model for quantitatively evaluating the output results of the multimodal large model; S4, evaluating the output results of the multimodal large model using the trained model for quantitatively evaluating the output results of the multimodal large model.

2. The multimodal large model quantitative evaluation method according to claim 1, characterized in that: The text pairs in the training set and the similarity values ​​corresponding to the text pairs are obtained by the following method: inputting a picture and a prompt text corresponding to the picture into a first multimodal large model, and the multimodal large model outputs a sixth text corresponding to the picture and the prompt text corresponding to the picture; inputting the picture and the prompt text corresponding to the picture into a second multimodal large model, and the multimodal large model outputs an eighth text corresponding to the picture and the prompt text corresponding to the picture; inputting the sixth text and the eighth text into a universal large model, and the universal large model outputs a similarity value between the sixth text and the eighth text; forming a text pair from the sixth text and the eighth text, and using the similarity value between the sixth text and the eighth text output by the universal large model as the similarity value corresponding to the text pair formed by the sixth text and the eighth text in the training set.

3. The multimodal large model quantitative evaluation method according to claim 1, characterized in that: The model for quantitatively evaluating the output results of the multimodal large model is constructed by taking the BERT model as the baseline model and adding a similarity task head and a similarity function; the text pair is input into the BERT model, the output of the BERT model is adjusted using the first method as the input of the similarity task head, the output of the similarity task head is used as the input of the similarity function, and the output of the similarity function is used as the output of the model for quantitatively evaluating the output results of the multimodal large model.

4. The multimodal large model quantitative evaluation method according to claim 3, characterized in that: The model for inputting text pairs into the output result of the multimodal large model is used to quantify and evaluate the output result of the multimodal large model, and the BERT model in the model for quantifying and evaluating the output result of the multimodal large model outputs a first feature matrix and a second feature matrix; The first feature matrix corresponds to one of the texts in a text pair input to a model for quantitatively evaluating an output result of a multimodal large model, and the second feature matrix corresponds to the other text in the text pair input to a model for quantitatively evaluating an output result of a multimodal large model.

5. The multimodal large model quantitative evaluation method according to claim 3, characterized in that: The first method refers to using the first row vector of the first feature matrix as the third feature vector corresponding to the first feature matrix, using the first row vector of the second feature matrix as the fourth feature vector corresponding to the second feature matrix, and using the third feature vector and the fourth feature vector as inputs of the similarity task head.

6. The multimodal large model quantitative evaluation method according to claim 3, characterized in that: The similarity task head is composed of a first linear layer, a first activation function, a second linear layer and an activation function; the input of the first linear layer is used as the input of the similarity task head, the output of the first linear layer is used as the input of the first activation function, the output of the first activation function is used as the input of the second linear layer, and the output of the second linear layer is used as the output of the similarity task head; The formula for the first linear layer is as follows: Linear1(x l1 )=W1x l1 +b1; Among them, Linear1(x l1 ) is the first linear layer; x l1 is the input of the first linear layer, W1 is the first network weight, and b1 is the first bias; The formula for the second linear layer is as follows: Linear2(x l2 )=W2x l2 +b2; Among them, Linear2(x l2 ) is the second linear layer; x l2 is the input of the second linear layer, W2 is the second network weight, and b2 is the second bias; The formula of the first activation function is as follows: Among them, GELU(x g1 ) is the first activation function; x g1 is the input of the first activation function, μ is the expectation, and δ is the standard deviation.

7. The multimodal large model quantitative evaluation method according to claim 3, characterized in that: The formula of the similarity function is as follows: Among them, f s is the similarity function, V i c is the i-th element in the fifth eigenvector; V i d is the i-th element in the sixth eigenvector.

8. The multimodal large model quantitative evaluation method according to claim 1, characterized in that: The model for quantitatively evaluating the output of the multimodal large model uses the following loss function: F l (and p ,and t )=(and p -and t ) 2 ; Among them, f l (y p ,y t ) is the loss function; y p is the similarity value of the model output for quantitatively evaluating the output result of the multimodal large model corresponding to the text pair input into the model for quantitatively evaluating the output result of the multimodal large model; y t It is the similarity value in the training set corresponding to the text pair input to the model for quantitatively evaluating the output results of the multimodal large model.

9. The multimodal large model quantitative evaluation method according to claim 1, characterized in that: The output result of the multimodal large model is evaluated by using the trained model for quantitatively evaluating the output result of the multimodal large model using the following method: Input a picture and a prompt text corresponding to the picture into the multimodal large model to be evaluated, and obtain the text output by the multimodal large model to be evaluated; a text pair is formed by the real label text corresponding to the picture and the prompt text corresponding to the picture and the text output by the multimodal large model to be evaluated, and the text pair is input into a trained model for quantitatively evaluating the output results of the multimodal large model; the trained model for quantitatively evaluating the output results of the multimodal large model outputs a similarity value corresponding to the text pair.

10. A multi-modal large model quantitative evaluation system, characterized in that: It includes data acquisition module, model building module, model testing module and result output module, among which: Data acquisition module, used to obtain training sets; A model building module, used to build a model for quantitatively evaluating the output results of a multimodal large model; A model testing module, used for training a model for quantitatively evaluating the output results of a multimodal large model using a training set obtained by the data acquisition module; The result output module is used to evaluate the output results of the multimodal large model using the trained model for quantitatively evaluating the output results of the multimodal large model.

Images