Model evaluation method based on multi-modal model

Abstract

The invention discloses a model evaluation method based on a multi-modal model. The model evaluation method is characterized by comprising the following steps: step 1, constructing a training data set; 2, building a multi-modal model according to the training data set built in the step 1; step 3, establishing a loss function which is designed according to different evaluation levels so as to ensure that the multi-modal model can be effectively trained on each level; and 4, inputting the multi-modal data into the large language model for model reasoning so as to generate a final evaluation result. According to the model evaluation method based on the multi-modal model, the model evaluation method based on the multi-modal model is provided so as to better evaluate the mastering and application ability of LLMs on scientific knowledge.

Description

Technical Field

[0001] The present invention relates to a model evaluation method, and more particularly to a model evaluation method based on a multimodal model. Background Art

[0002] With the rapid development of artificial intelligence technology, especially the emergence and application of large language models (LLMs), the evaluation of scientific intelligence has become an important research topic. Existing evaluation methods mainly focus on single-dimensional knowledge coverage or simple task execution capabilities, lacking a multi-level and comprehensive evaluation of the model's scientific knowledge. Summary of the invention

[0003] In view of the shortcomings of the prior art, the purpose of the present invention is to provide a model evaluation method based on a multimodal model, which is particularly important for better evaluating the mastery and application ability of LLMs in scientific knowledge.

[0004] To achieve the above object, the present invention provides the following technical solution: a model evaluation method based on a multimodal model, characterized in that it comprises the following steps:

[0005] Step 1: Build a training data set;

[0006] Step 2: Build a multimodal model based on the training data set constructed in step 1;

[0007] Step 3: Establish a loss function, which is designed according to different evaluation levels to ensure that the multimodal model can be effectively trained at all levels;

[0008] Step 4: Input the multimodal data into the large language model for model inference to generate the final evaluation results.

[0009] As a further improvement of the present invention, the training data set constructed in step 1 includes the following five levels:

[0010] Broad understanding layer: data includes basic concepts, definitions, theories and common sense of different scientific disciplines;

[0011] In-depth exploration layer: data includes the scientific research process, experimental methods, data analysis and research results; Deep reasoning layer: data includes the reasoning process, logical relationships, mathematical formulas and calculation steps of scientific problems; Accurate assessment layer: data should include scientific ethical cases, safety assessment standards, and information identification methods;

[0012] Practical application layer: data includes solutions to practical problems, innovative design ideas, and engineering implementation methods.

[0013] As a further improvement of the present invention, the multimodal model constructed in step 2 includes:

[0014] Encoders, used to process data of different modalities;

[0015] The fusion layer is used to fuse data from different modalities to generate a unified representation;

[0016] Decoders, which are used to generate outputs such as answering questions and generating text.

[0017] As a further improvement of the present invention, the encoder adopts the Transformer architecture, BERT is used for text encoder output, and VisionTransformer is used for image encoder output; the fusion layer adopts the attention mechanism method to fuse data; and the decoder adopts the Transformer architecture.

[0018] As a further improvement of the present invention, the variable name and formula of the multimodal model constructed in step 2 are as follows: Text encoder output: Ht = BERT (Xt)

[0019] Image encoder output: Hi = ViT (Xi)

[0020] Fusion representation: Hf = Fusion (Ht, Hi)

[0021] Decoder output: Y = Decoder (Hf).

[0022] As a further improvement of the present invention, the loss function established in step 3 includes:

[0023] Broadly understand the loss function, cross entropy loss, for classification and recognition of basic knowledge, the expression is as follows:

[0024]

[0025] Deeper exploration of loss: The loss function based on information retrieval and question answering is expressed as follows:

[0026]

[0027] Deep reasoning loss: The loss function of logical reasoning and numerical calculation is expressed as follows:

[0028]

[0029] Accurately assess the loss: The loss function involving ethical and safety assessment is expressed as follows:

[0030]

[0031] Practical application loss: The loss function for practical problem solving and innovative design is expressed as follows:

[0032]

[0033] As a further improvement of the present invention, the specific steps of model reasoning in step 4 are as follows:

[0034] Step 41: preprocess the multimodal data to generate an input format acceptable to the model;

[0035] Step 42: Generate output through encoder, fusion layer and decoder to complete model reasoning;

[0036] Step 43: Post-process the model output results to generate the final evaluation results. The specific steps include formatting the generated text and verifying the prediction results.

[0037] As a further improvement of the present invention, the formula for preprocessing the multimodal data in step 41 is as follows:

[0038] X t =Preprocess(text)

[0039] X i =Preprocess(image).

[0040] As a further improvement of the present invention, the formula for generating output by the encoder, fusion layer and decoder in step 42 is as follows:

[0041] H t =BERT(X t )

[0042] H i =ViT(X i

[0043] H f =Fusion(H t ,H i )

[0044] Y=Decoder(H f ).

[0045] Beneficial effects of the present invention:

[0046] Comprehensively evaluate the scientific knowledge of the model. By constructing a training data set containing five levels, the present invention can comprehensively evaluate the knowledge coverage and depth of the multimodal model in the scientific field. Each level of the data set targets different scientific knowledge and application scenarios, ensuring that the model can be fully trained and evaluated in terms of broad understanding, in-depth exploration, deep reasoning, accurate evaluation and practical application. Improve the multimodal fusion capability of the model; the model structure designed by the present invention includes an encoder, a fusion layer and a decoder, which can process and fuse multiple data types (text, images, videos, etc.). By fusing data from different modalities, the model can generate richer and more accurate representations, improving the understanding and reasoning capabilities of complex scientific problems. DETAILED DESCRIPTION

[0047] The present invention will be further described in detail with reference to the given embodiments below.

[0048] A model evaluation method based on a multimodal model in this embodiment specifically includes the following steps: Step 1: Construct a training data set

[0049] Constructing a training dataset is a key step in achieving scientific knowledge assessment. The training dataset should include text, images, videos, and other multimodal data from a variety of scientific fields. The dataset should cover the following five levels: General Comprehension: Collect a wide range of scientific literature, textbooks, encyclopedias, etc., covering basic knowledge in multiple scientific fields. The data should include basic concepts, definitions, theories, and common sense in different scientific disciplines.

[0050] In-depth Exploration: Including scientific research papers, experimental reports, scientific discussions, etc., with the focus on in-depth exploration and analysis of scientific issues. Data should include the scientific research process, experimental methods, data analysis and research results.

[0051] Deep Reasoning: includes scientific reasoning questions, logical reasoning questions, mathematical calculation questions, etc., requiring the model to perform complex logical reasoning and numerical calculations. The data should include the reasoning process, logical relationships, mathematical formulas, and calculation steps of scientific problems.

[0052] Accurate Assessment: This involves scientific ethics, safety assessment, information identification, etc., and requires the model to make accurate and ethical judgments. The data should include scientific ethics cases, safety assessment standards, information identification methods, etc.

[0053] Practical Application: including solutions to actual scientific problems, innovative designs, engineering applications, etc., to evaluate the model’s ability to be applied in the real world. The data should include solutions to actual problems, innovative design ideas, engineering implementation methods, etc.

[0054] Step 2: Build the model structure

[0055] The multimodal model structure should be able to process and fuse multiple data types (text, images, videos, etc.) in order to comprehensively evaluate its scientific knowledge and application capabilities. The model structure includes the following parts:

[0056] Encoder: used to process data of different modalities. Transformer architecture can be used, such as BERT for text and Vision Transformer (ViT) for images.

[0057] Fusion Layer: Fusions data from different modalities to generate a unified representation. This can be done using attention mechanisms or other fusion methods.

[0058] Decoder: used to generate output, such as answering questions, generating text, etc. The decoder can use the Transformer architecture or other generative models.

[0059] The specific variable names and formulas are as follows:

[0060] Text encoder output: Ht = BERT(Xt)

[0061] Image encoder output: Hi = ViT (Xi)

[0062] Fusion representation: Hf = Fusion (Ht, Hi)

[0063] Decoder output: Y = Decoder (Hf)

[0064] Step 3: Establish the loss function

[0065] The loss function should be designed according to different evaluation levels to ensure that the model can be effectively trained at all levels.

[0066] General Comprehension Loss: Cross entropy loss for classification and recognition of basic knowledge.

[0067]

[0068] In-depthExplorationLoss: A loss function based on information retrieval and question answering.

[0069]

[0070] DeepReasoningLoss: loss function for logical reasoning and numerical calculation.

[0071]

[0072] AccurateAssessmentLoss: A loss function involving ethical and safety assessments.

[0073]

[0074] Practical Application Loss: Loss function for practical problem solving and innovative design.

[0075]

[0076] Step 4: Perform model inference

[0077] The model inference process includes the following steps:

[0078] Input data preprocessing: Preprocess multimodal data (text, images, etc.) to generate an input format acceptable to the model.

[0079] X t =Preprocess(text)

[0080] X i =Preprocess(image)

[0081] Model inference: Generate output through encoder, fusion layer and decoder.

[0082] H t =BERT(X t )

[0083] H i =ViT(X i

[0084] H f =Fusion(H t ,H i )

[0085] Y=Decoder(H f )

[0086] Output result post-processing: Post-process the model output results to generate the final evaluation results. Specific steps include formatting the generated text, verifying the prediction results, etc.

[0087] This embodiment provides the following practical use examples:

[0088] Building a training dataset

[0089] A small-scale multimodal dataset was collected for training and evaluating the model. The dataset includes the following:

[0090] General Comprehension: Contains text data from 100 science textbook chapters.

[0091] In-depth Exploration: Contains summaries of 50 scientific research papers.

[0092] Deep Reasoning: Contains 200 scientific reasoning questions.

[0093] Accurate Assessment: Contains 30 scientific ethics cases.

[0094] Practical Application: Contains descriptions of solutions to 20 practical scientific problems.

[0095] Model structure

[0096] Text Encoder: Use the BERT model to process text data.

[0097] Image Encoder: Use the ResNet model to process image data.

[0098] Fusion Layer: Fusions the text and image representations together.

[0099] Decoder: Generates the output using a simple feed-forward neural network.

[0100] Loss Function

[0101] GeneralComprehensionLoss: Use cross entropy loss to evaluate the model’s understanding of basic knowledge.

[0102] In-depthExplorationLoss: Use mean squared error loss to evaluate the model's answers to scientific research questions.

[0103] DeepReasoningLoss: Use mean squared error loss to evaluate the logical reasoning and computational capabilities of the model.

[0104] AccurateAssessmentLoss: Use cross entropy loss to evaluate the model’s judgment on ethical and safety issues.

[0105] Practical Application Loss: Use mean squared error loss to evaluate the model’s generation of solutions to practical problems.

[0106] Model Inference

[0107] Just perform normal model reasoning according to the reasoning process in the content of the invention.

[0108] The above is only a preferred embodiment of the present invention, and the protection scope of the present invention is not limited to the above embodiments. All technical solutions under the concept of the present invention belong to the protection scope of the present invention. It should be pointed out that for ordinary technicians in this technical field, some improvements and modifications without departing from the principle of the present invention should also be regarded as the protection scope of the present invention.

Claims

1. A model evaluation method based on a multimodal model, characterized in that: The steps include: Step 1: Build a training data set; Step 2: Build a multimodal model based on the training data set constructed in step 1; Step 3: Establish a loss function, which is designed according to different evaluation levels to ensure that the multimodal model can be effectively trained at all levels; Step 4: Input the multimodal data into the large language model for model inference to generate the final evaluation results.

2. The model evaluation method based on a multimodal model according to claim 1, characterized in that: The training data set constructed in step 1 includes the following five levels: Broad understanding layer: data includes basic concepts, definitions, theories and common sense of different scientific disciplines; In-depth exploration layer: data includes the scientific research process, experimental methods, data analysis and research results; Deep reasoning layer: data includes the reasoning process, logical relationships, mathematical formulas and calculation steps of scientific problems; Accurate assessment layer: data should include scientific ethics cases, safety assessment standards, and information identification methods; Practical application layer: data includes solutions to practical problems, innovative design ideas, and engineering implementation methods.

3. The model evaluation method based on a multimodal model according to claim 1 or 2, characterized in that: The multimodal model constructed in step 2 includes: Encoders, used to process data of different modalities; The fusion layer is used to fuse data from different modalities to generate a unified representation; Decoders, which are used to generate outputs such as answering questions and generating text.

4. The model evaluation method based on a multimodal model according to claim 3, characterized in that: The encoder adopts the Transformer architecture, BERT is used for text encoder output, and Vision Transformer is used for image encoder output; the fusion layer adopts the attention mechanism method to fuse data; the decoder adopts the Transformer architecture.

5. The model evaluation method based on a multimodal model according to claim 4, characterized in that: The variable names and formulas of the multimodal model constructed in step 2 are as follows: Text encoder output: Ht = BERT(Xt) Image encoder output: Hi = ViT (Xi) Fusion representation: Hf = Fusion (Ht, Hi) Decoder output: Y = Decoder (Hf).

6. The model evaluation method based on a multimodal model according to claim 1 or 2, characterized in that: The loss function established in step 3 includes: Broadly understand the loss function, cross entropy loss, for classification and recognition of basic knowledge, the expression is as follows: Deeper exploration of loss: The loss function based on information retrieval and question answering is expressed as follows: Deep reasoning loss: The loss function of logical reasoning and numerical calculation is expressed as follows: Accurately assess the loss: The loss function involving ethical and safety assessment is expressed as follows: Practical application loss: The loss function for practical problem solving and innovative design is expressed as follows:

7. The model evaluation method based on a multimodal model according to claim 1 or 2, characterized in that: The specific steps of model reasoning in step 4 are as follows: Step 41: preprocess the multimodal data to generate an input format acceptable to the model; Step 42: Generate output through encoder, fusion layer and decoder to complete model reasoning; Step 43: Post-process the model output results to generate the final evaluation results. The specific steps include formatting the generated text and verifying the prediction results.

8. The model evaluation method based on a multimodal model according to claim 7, characterized in that: The formula for preprocessing multimodal data in step 4-1 is as follows: X t =Preprocess9textZ X i =Preprocess(image)。 9. The model evaluation method based on a multimodal model according to claim 8, characterized in that: The formula for generating outputs by the encoder, fusion layer, and decoder in step 42 is as follows: H t =BERT(X t ) H i =ViT(X i ) H f =Fusion(H t ,H i ) Y=Decoder(H f )。

Images