Modularized large language model evaluation method and device, and medium

Abstract

The invention discloses an evaluation method and device for a modular large language model and a medium, and relates to the technical field of artificial intelligence. The method comprises the steps of receiving original input data containing at least one mode, and performing structure analysis and field extraction on the original input data according to a predefined task type to generate a standard structured input sample; according to the task type, dynamically matching and scheduling a corresponding multi-modal large language model from a registered model service to obtain original output content; performing structure extraction and semantic regularization processing on the original output content to obtain regularized answer content; calling a corresponding automatic evaluation algorithm to calculate the normalized answer content, scoring through a preset auxiliary scoring big language model, and generating an automatic index score and a subjective index score; and based on the automatic index score and the subjective index score, generating a structured evaluation report containing visual content.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of artificial intelligence, and particularly relates to a modular large language model evaluation method, device and medium. BACKGROUND

[0002] In recent years, with the rapid development of artificial intelligence technology, large language models have gradually evolved from pure text processing to important tools capable of understanding and generating multi-modal content such as images and videos. Multi-modal large language models have shown wide application potential in image description, video question answering, and image-text reasoning tasks. However, the existing large model evaluation system still mainly focuses on text tasks, and its evaluation indicators, data interfaces and task paradigms are difficult to directly migrate to the multi-modal scene, resulting in limitations on the comprehensiveness and reliability of the evaluation results.

[0003] Current mainstream evaluation tools are mostly based on static data sets and fixed evaluation processes, and lack unified processing capabilities for multi-modal inputs. Different modal data has significant differences in representation form, semantic granularity, evaluation criteria, etc., and existing methods often cannot achieve cross-modal semantic alignment and consistency evaluation. In addition, these tools usually tightly couple models, tasks and evaluation indicators, resulting in poor system scalability, high cost of adapting to new tasks, and difficulty in meeting the rapid iteration needs of model development.

[0004] Therefore, how to construct a modular, scalable, unified indicator and semantic understanding evaluation framework under the premise of supporting multi-modal input has become a technical problem that needs to be solved by those skilled in the art. SUMMARY

[0005] The embodiments of the present application provide a modular large language model evaluation method, device and medium, to solve the technical problem of how to construct a modular, scalable, unified indicator and semantic understanding evaluation framework under the premise of supporting multi-modal input.

[0006] In a first aspect, the embodiments of the present application provide an evaluation method of a modular large language model, which comprises: receiving original input data containing at least one modality, and performing structural analysis and field extraction on the original input data according to a predefined task type to generate a standard structured input sample; dynamically matching and scheduling a corresponding multi-modal large language model from a registered model service according to the task type, and sending the structured input sample to the multi-modal large language model for inference to obtain original output content; performing structural extraction and semantic regularization processing on the original output content to obtain regularized answer content; according to the task type, calling a corresponding automatic evaluation algorithm to calculate the regularized answer content to generate an automatic index score, and constructing a prompt word containing the regularized answer content and an evaluation instruction to score through a preset auxiliary scoring large language model to generate a subjective index score; and generating a structured evaluation report containing visualized content based on the automatic index score and the subjective index score.

[0007] In an embodiment of the present application, the original input data is analyzed and field extracted according to the predefined task type to generate a standard structured input sample, specifically comprising: identifying the task type annotated by the original input data; extracting the corresponding key fields from the original input data according to the task type; wherein the key fields include at least one of question text, reference answer text, image path, and video frame path set; and filling the key fields into a predefined standardized template to form a structured input sample.

[0008] In an embodiment of the present application, the multi-modal large language model is dynamically matched and scheduled from the registered model service according to the task type, specifically comprising: querying a pre-configured task-model mapping table to obtain one or more preferred model identifiers corresponding to the task type of the original input data; calling the model service corresponding to the preferred model identifier in priority order; and if the model service with the highest priority fails to be called, automatically calling an alternative model service.

[0009] In an embodiment of the present application, the original output content is structurally extracted, specifically comprising: selecting a corresponding structural extraction strategy from a predefined rule library according to the task type; and extracting core semantic content from the original output content according to the selected structural extraction strategy, and excluding preset template statements and redundant information.

[0010] In an embodiment of the present application, the original output content is subjected to semantic normalization processing, specifically including: performing normalization processing on the case and punctuation of the text in the original output content; performing standardized conversion on the physical units mentioned in the text; extracting keywords in the original output content using an unsupervised keyword extraction algorithm, and calculating the importance score of each sentence in the original output content based on the textrank algorithm; eliminating redundant sentences based on a sentence similarity threshold, and after eliminating the redundant sentences, retaining the first pre-set number of sentences according to the importance score of the sentences.

[0011] In an embodiment of the present application, according to the task type, a corresponding automatic evaluation algorithm is called to calculate the normalized answer content, specifically including: if the task type is a text generation task, calculating the text similarity index between the normalized answer content and the reference answer based on n-gram matching; if the task type involves image modalities, a visual language joint embedding model calculates the semantic similarity between the normalized answer content and the reference image; if the task type involves video modalities, the semantic aggregation result of the video frames is calculated.

[0012] In an embodiment of the present application, a prompt word containing the normalized answer content and evaluation instructions is constructed to score through a pre-set auxiliary scoring large language model, specifically including: constructing an evaluation instruction template; wherein the evaluation instruction template contains multiple subjective dimensions to be evaluated, original input data, normalized answer content, and optional reference answers; sending the evaluation instruction template to the auxiliary scoring large language model and requesting the auxiliary scoring large language model to return the scoring results of each dimension in a structured data format; performing format verification on the returned scoring results.

[0013] In an embodiment of the present application, based on the automatic index score and the subjective index score, a structured evaluation report containing visual content is generated, specifically including: fusing the automatic index score and the subjective index score according to a pre-defined weight to obtain a comprehensive evaluation score; generating visual charts containing index comparison, semantic heat map and weak item analysis according to the task type and the comprehensive evaluation score; combining the visual charts, the normalized answer content and the original input data according to a pre-set template, and outputting a structured evaluation report in at least one format.

[0014] In a second aspect, the embodiments of the present application also provide an evaluation device of a modular large language model, the device comprising: at least one processor; and a memory communicatively connected with the at least one processor; wherein the memory stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor to enable the at least one processor to perform any one of the above modular large language model evaluation methods.

[0015] In a third aspect, the embodiments of the present application further provide a non-volatile computer storage medium, which stores computer executable instructions. When the computer executable instructions are executed, a method for evaluating a modular large language model according to any one of the preceding aspects is implemented.

[0016] The method for evaluating a modular large language model, the device and the medium provided by the embodiments of the present application have the following beneficial effects: through the modular architecture and the multi-modal unified processing mechanism, the expansibility, accuracy and practicability of the large language model evaluation are significantly improved. The method can flexibly adapt to various modal inputs and task types such as text, image and video, and effectively overcomes the limitations of traditional evaluation tools in cross-modal semantic alignment and output noise suppression through structured data representation, dynamic model scheduling and semantic regularization processing. At the same time, by combining automatic indicators and subjective scores based on large language models, a multi-dimensional and interpretable comprehensive evaluation system is constructed, which not only enhances the comprehensiveness and reliability of the evaluation results, but also provides efficient and reliable support for model optimization, technical review and application delivery through the functions of visual report and historical comparison. BRIEF DESCRIPTION OF DRAWINGS

[0017] The accompanying drawings described herein are used to provide further understanding of the present application, form a part of the present application, and the illustrative embodiments of the present application and their descriptions are used to explain the present application, and do not constitute improper limitations on the present application. In the drawings: Figure 1 A flowchart of a method for evaluating a modular large language model is provided for the embodiments of the present application; Figure 2 An internal structure schematic diagram of an evaluation device for a modular large language model is provided for the embodiments of the present application. DETAILED DESCRIPTION

[0018] In order to make the purpose, technical scheme and advantages of the present application clearer, the technical scheme of the present application will be described in detail below in combination with the embodiments of the present application and the corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of the present application, not all the embodiments. Based on the embodiments in the present application, all other embodiments obtained by those skilled in the art without creative labor fall within the scope of protection of the present application.

[0019] The embodiments of the present application provide a method for evaluating a modular large language model, a device and a medium, to solve the technical problem of how to construct a modular, scalable, unified indicator and semantic understanding evaluation framework under the premise of supporting multi-modal input.

[0020] The technical scheme proposed by the embodiments of the present application will be described in detail below with reference to the drawings.

[0021] Figure 1 A flowchart of an evaluation method of a modular large language model is provided for an embodiment of the present application. As shown in Figure 1 The evaluation method of a modular large language model provided by the present application specifically includes the following steps: Step 101, receiving original input data containing at least one modality, and performing structural analysis and field extraction on the original input data according to a predefined task type to generate a standard structured input sample.

[0022] In an embodiment of the present application, the original input data is analyzed and extracted according to the predefined task type to generate a standard structured input sample, specifically including: identifying the task type annotated by the original input data; extracting the corresponding key fields from the original input data according to the task type; wherein the key fields include at least one of question text, reference answer text, image path, and video frame path set; filling the key fields into a predefined standardized template to form a structured input sample.

[0023] In this embodiment, the “original input data” refers to a sample instance to be evaluated, which can include text, image, audio, video or any combination thereof, and is usually derived from a public evaluation dataset, user real-time upload or a data pool collected in a specific business scenario. The “predefined task type” is a type of evaluation task declared by the system in advance, such as visual question answering (VQA), image captioning, video summarization or text generation, etc. This type can be indicated by the metadata label of the input data, or automatically inferred and identified by the system according to the input content.

[0024] Further, the "structural analysis and field extraction" refers to extracting key information units necessary for reasoning and evaluation from the original data according to the definition of the current task type. For example, for a visual question answering task (task_type:vqa), this process will parse and extract the "question text" (question), "image storage path or encoded image data" (image_path or image_data), and optional "reference answer" (reference_answer) fields from the original data; while for a video understanding task, the "question text" and "video file path or key frame sequence" fields need to be extracted. Subsequently, these extracted key fields are filled into a "standardized template", which is a pre-designed data structure that can be in the form of a JSON object, Python dictionary or other forms of intermediate representation. Its role is to ensure that regardless of the form of the original data, the "structured input sample" generated finally follows a uniform internal schema, thereby providing a consistent data interface for subsequent steps of model invocation, Prompt construction and evaluation comparison. The system also supports dynamically declaring new task types and their required field mapping rules through configuration files (such as YAML or JSON files), thereby realizing flexible expansion of processing capabilities without the need to modify the core code to adapt to new tasks.

[0025] Step 102, according to the task type, dynamically match and schedule the corresponding multi-modal large language model from the registered model service, and send the structured input sample to the multi-modal large language model for reasoning to obtain the original output content.

[0026] In an embodiment of the present application, according to the task type, the corresponding multi-modal large language model is dynamically matched and scheduled from the registered model service, specifically including: querying the pre-configured task-model mapping table to obtain one or more preferred model identifiers corresponding to the task type of the original input data; calling the model service corresponding to the preferred model identifier in priority order; if the highest priority model service fails to call, automatically downgrade to call the alternative model service.

[0027] In this embodiment, "registered model service" refers to those model instances that have completed registration and encapsulation within the system. They can be locally deployed (such as using LLaVA, BLIP-2, etc. open source models), or they can be cloud APIs (such as GPT-4V, Claude 3, etc.). All models provide services through a unified interface specification (such as HTTP / REST or gRPC), and their meta information (such as model identifier, service endpoint, capability description, priority, etc.) is recorded in a centralized model registry (such as implemented using Redis, MongoDB or Etcd).

[0028] Further, "dynamic matching and scheduling" refers to querying a pre-configured "task-model mapping table" according to the task_type determined in step 101, which defines the list of models most suitable for use for different task types and their call priority. For example, the mapping table can be configured as: when the task_type is image_captioning, the blip2_vicuna model is preferentially called, and if the service is unavailable or times out, it is automatically downgraded (fallback) to llava_1.5 or gpt4v. This scheduling mechanism also allows the integration of more complex strategies, such as switching between precision-first and efficiency-first modes to adapt to the needs of different evaluation scenarios. After matching to the appropriate model service, the system encapsulates the structured input sample into the input format required by the model (for example, for image-text models, it may be necessary to convert the image path into an image embedding, or to splice the image and text into a specific Prompt format), and then initiates an inference request through efficient mechanisms such as asynchronous calls, batch processing, etc. Finally, the "raw output content" returned by the model is received and temporarily stored, which is usually unprocessed, directly generated text or structured response from the model.

[0029] Step 103, performing structural extraction and semantic regularization processing on the raw output content to obtain the regularized answer content.

[0030] In an embodiment of the present application, the structural extraction of the raw output content specifically includes: selecting the corresponding structural extraction strategy from the pre-defined rule library according to the task type; extracting the core semantic content from the raw output content according to the selected structural extraction strategy, and excluding the pre-set template sentences and redundant information.

[0031] In an embodiment of the present application, the semantic regularization processing of the raw output content specifically includes: performing normalization processing on the case and punctuation of the text in the raw output content; performing standardized conversion on the physical units mentioned in the text; using an unsupervised keyword extraction algorithm to extract keywords from the raw output content, and calculating the importance score of each sentence in the raw output content based on the textrank algorithm; performing redundant sentence elimination based on the sentence similarity threshold, and after eliminating the redundant sentences, retaining the first pre-set number of sentences according to the importance score of the sentences.

[0032] In this embodiment, "structure extraction" is the first step of processing for the original output content, and its strategy is closely related to the current task type. The system maintains a "task type-extraction strategy" rule library, for example: for the "single-turn question answering" task, the strategy may be to extract the first sentence in the output or the sentence that directly answers the question; for the "multi-turn dialogue" task, the response part for the last question is extracted; for the "image description" task, focus on the sentence describing the main body and key attributes of the image, excluding introductory sentences such as "This picture shows…".

[0033] Further, after completing the preliminary extraction, "semantic regularization" further cleanses and standardizes the text. This includes: text normalization (such as converting to all lowercase, full / half-width symbol conversion), physical unit standardization (such as converting "one meter" and "1 meter" to "1m", which can be achieved with tools such as Quantulum3), and using keyword extraction techniques based on TextRank or BERT models to identify and remove sentences with redundant semantics or low information content, thereby compressing the content and retaining the essence. For complex outputs such as code generation, regularization may also include code structure parsing and standardization using AST (Abstract Syntax Tree). This step finally produces "regularized answer content", which is a pure, concise, and standardized text string, highly focused on the core answer required by the task, greatly improving the accuracy and consistency of subsequent evaluation metrics.

[0034] Step 104, according to the task type, call the corresponding automatic evaluation algorithm to calculate the regularized answer content, generate automatic index score, and build prompt words containing regularized answer content and evaluation instructions, to score through the preset auxiliary scoring large language model, generate subjective index score.

[0035] In an embodiment of the present application, according to the task type, the corresponding automatic evaluation algorithm is called to calculate the regularized answer content, specifically including: if the task type is a text generation task, calculate the text similarity index between the regularized answer content and the reference answer based on n-gram matching; if the task type involves image modalities, a visual language joint embedding model calculates the semantic similarity between the regularized answer content and the reference image; if the task type involves video modalities, the semantic aggregation results of the video frames are calculated.

[0036] In an embodiment of the present application, a prompt word containing the regularized answer content and the evaluation instruction is constructed to score by a preset auxiliary scoring large language model, specifically including: constructing an evaluation instruction template; wherein the evaluation instruction template contains a plurality of subjective dimensions to be evaluated, original input data, regularized answer content, and optional reference answers; sending the evaluation instruction template to the auxiliary scoring large language model and requesting the auxiliary scoring large language model to return the scoring results of each dimension in a structured data format; and performing format checking on the returned scoring results.

[0037] In the present embodiment, according to the task type, the corresponding automatic evaluation algorithm is called to calculate the regularized answer content to generate an automatic index score; at the same time, a prompt word (Prompt) containing the regularized answer content, the original question and the evaluation instruction is constructed, and the subjective index score is generated by scoring through a preset auxiliary scoring large language model (such as GPT-4).

[0038] It can be understood that the present step is the core of the evaluation, which realizes the multi-dimensional fusion of objective indexes and subjective evaluation. The selection of the "automatic evaluation algorithm" depends on the nature of the task: for text generation tasks, the system will calculate the classical indexes between the regularized answer and the reference answer, such as BLEU, ROUGE based on n-gram overlap, or F1Score, Exact Match based on token matching; for tasks involving visual modalities (such as image-text question answering), a visual-language joint embedding model such as CLIP will be called to calculate the similarity between the generated text and the reference image in the semantic space as a consistency measure; for video tasks, the score may be based on the semantic aggregation results of video frames or transcription captions.

[0039] At the same time, the "auxiliary scoring large language model" is introduced to evaluate the subjective dimensions that are difficult for machines to quantify. The system will carefully construct an evaluation instruction template (Prompt), which clearly specifies the dimensions to be scored (such as logic, conciseness, factual accuracy, language quality, etc.), and provides "questions", "model-generated answers" (i.e. regularized content) and optional "reference answers" to be evaluated. The Prompt is sent to large language models such as GPT-4, and is required to return the scores of each dimension in strict JSON format. By setting low temperature (temperature=0) and few-shot examples, the stability and consistency of the scoring results are guaranteed. The returned JSON result will be format checked to ensure its usability. This makes up for the shortcomings of pure automatic indexes in measuring the quality of answers, logical coherence, etc.

[0040] Step 105, based on the automatic index score and the subjective index score, a structured evaluation report containing visual content is generated.

[0041] In an embodiment of the present application, based on the automatic indicator score and the subjective indicator score, a structured evaluation report containing visual content is generated, specifically including: fusing the automatic indicator score and the subjective indicator score according to a predefined weight to obtain a comprehensive evaluation score; generating visual charts containing indicator comparison, semantic heat map and weak item analysis according to the task type and the comprehensive evaluation score; combining the visual charts, the regularized answer content and the original input data according to a preset template, and outputting a structured evaluation report in at least one format.

[0042] In this embodiment, this step is the final stage of the evaluation process, aiming to synthesize, present and deliver the evaluation results. The system provides multiple score fusion strategies, such as assigning different static weights to the automatic indicator score and the subjective indicator score for weighted summation, or using more complex Bayesian optimization methods to dynamically learn the optimal weight combination to minimize the difference with human evaluation, and finally obtain a "comprehensive evaluation score".

[0043] Subsequently, the system automatically generates rich "visual content" according to the task type and all score data. These contents can include but are not limited to: radar chart or column chart comparison of each indicator, semantic similarity heat map of model output and reference answer, and analysis text based on score weak items. All these elements, including original input, regularized output, score details of each item and visual charts, are finally combined and filled into a "preset template" to output a structured evaluation report. The report supports multiple practical formats, such as JSON for further processing by programs, CSV for data recording, and PDF or HTML documents for easy human reading with illustrations. The report provides intuitive, comprehensive and reliable decision-making basis for model performance comparison, iteration optimization and project delivery.

[0044] The above is an embodiment of the method of the present application. Based on the same inventive concept, the embodiments of the present application also provide an evaluation device for a modular large language model, the structure of which is as shown in Figure 2 .

[0045] Figure 2 An internal structure diagram of an evaluation device for a modular large language model provided by an embodiment of the present application is shown in Figure 2 . The device includes: at least one processor 201; and a memory 202 in communication connection with the at least one processor; Wherein the memory 202 stores instructions executable by the at least one processor, and the instructions are executed by the at least one processor 201 to enable the at least one processor 201 to: The original input data containing at least one mode is received, and the original input data is structurally parsed and field extracted according to a predefined task type to generate a standard structured input sample; according to the task type, a corresponding multi-modal large language model is dynamically matched and scheduled from a registered model service, and the structured input sample is sent to the multi-modal large language model for inference to obtain original output content; the original output content is subjected to structure extraction and semantic regularization processing to obtain regularized answer content; according to the task type, a corresponding automatic evaluation algorithm is called to calculate the regularized answer content to generate an automatic index score, and a prompt word containing the regularized answer content and evaluation instructions is constructed to score through a preset auxiliary scoring large language model to generate a subjective index score; based on the automatic index score and the subjective index score, a structured evaluation report containing visualized content is generated.

[0046] Some embodiments of the present application provide a non-volatile computer storage medium corresponding to Figure 1 a non-volatile computer storage medium, storing computer executable instructions, the computer executable instructions being configured to: receive original input data containing at least one mode, and perform structural parsing and field extraction on the original input data according to a predefined task type to generate a standard structured input sample; according to the task type, a corresponding multi-modal large language model is dynamically matched and scheduled from a registered model service, and the structured input sample is sent to the multi-modal large language model for inference to obtain original output content; the original output content is subjected to structure extraction and semantic regularization processing to obtain regularized answer content; according to the task type, a corresponding automatic evaluation algorithm is called to calculate the regularized answer content to generate an automatic index score, and a prompt word containing the regularized answer content and evaluation instructions is constructed to score through a preset auxiliary scoring large language model to generate a subjective index score; based on the automatic index score and the subjective index score, a structured evaluation report containing visualized content is generated.

[0047] Each of the embodiments in the present application is described in a progressive manner, and the same or similar parts between the embodiments can be referred to each other. Each embodiment focuses on the difference from other embodiments. In particular, for the Internet of Things device and medium embodiments, since they are basically similar to the method embodiments, the description is relatively simple, and the relevant parts can be referred to the part of the method embodiment.

[0048] The system and medium provided by the embodiments of the present application are one-to-one corresponding to the method, so the system and medium also have similar beneficial technical effects as the method. Since the beneficial technical effects of the method have been described in detail above, the beneficial technical effects of the system and medium will not be described here.

[0049] Those skilled in the art will appreciate that embodiments of the present application can be readily used as software, hardware, or a combination of software and hardware. In a typical configuration, a computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0050] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for functionally implementing one or more functions specified in the flowchart block or blocks.

[0051] These computer program instructions can also be stored in a computer readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer readable memory produce an article of manufacture including instructions which implement the function specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for functionally implementing one or more functions specified in the flowchart block or blocks.

[0052] The computer program instructions can also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart block or blocks. Figure 1 one or more functions specified in the flowchart block or blocks. Figure 1 means for functionally implementing one or more functions specified in the flowchart block or blocks.

[0053] In one typical configuration, the computing device includes one or more processors (CPUs), input / output interfaces, network interfaces, and memory.

[0054] The memory can include non-persistent memory and / or persistent memory, such as flash memory, read-only memory (ROM), and / or the like, in a computer readable medium. The memory is an example of computer readable media.

[0055] Computer-readable media includes permanent and non-permanent, movable and non-movable media that can be implemented by any method or technology to store information. The information can be computer-readable instructions, data structures, program modules or other data. Examples of computer storage media include, but are not limited to, phase-change memory (PRAM), static random access memory (SRAM), dynamic random access memory (DRAM), other types of random access memory (RAM), read-only memory (ROM), electrically erasable programmable read-only memory (EEPROM), flash memory or other memory technology, compact disc read-only memory (CD-ROM), digital versatile disc (DVD) or other optical storage, magnetic cassette, magnetic tape, magnetic disk storage or other magnetic storage devices, or any other non-transmission medium that can be used to store information accessible to a computing device. According to the definition herein, computer-readable media does not include transitory media such as modulated data signals and carriers.

[0056] It should also be noted that the terms "comprising", "containing", or any other variant thereof are intended to cover non-exclusive inclusions, so that a process, method, article or apparatus that includes a list of elements does not only include those elements, but also includes other elements not explicitly listed, or further includes elements inherent in such a process, method, article or apparatus. Without more limitations, the element defined by the statement "comprising a" does not exclude the presence of additional identical elements in the process, method, article or apparatus that includes the element.

[0057] The above only describes the embodiments of the present application and is not intended to limit the present application. For those skilled in the art, the present application can have various modifications and changes. Any modification, equivalent replacement, improvement, etc. within the spirit and principle of the present application shall be included in the scope of claims of the present application.

Claims

1. An evaluation method for a modular large language model, characterized in that, The method includes: Receive raw input data containing at least one modality, and perform structural parsing and field extraction on the raw input data according to a predefined task type to generate a standard structured input sample; Based on the task type, the corresponding multimodal large language model is dynamically matched and scheduled from the registered model services, and the structured input sample is sent to the multimodal large language model for inference to obtain the original output content; The original output content is subjected to structural extraction and semantic normalization to obtain the normalized answer content; According to the task type, the corresponding automatic evaluation algorithm is invoked to calculate the normalized answer content, generate an automatic index score, and construct prompt words containing the normalized answer content and evaluation instructions, so as to score through a preset auxiliary scoring large language model and generate a subjective index score. Based on the automatic indicator scores and the subjective indicator scores, a structured evaluation report containing visualization content is generated.

2. The evaluation method for a modular large language model according to claim 1, characterized in that, Based on a predefined task type, the original input data undergoes structural parsing and field extraction to generate standard structured input samples, specifically including: Identify the task type labeled in the original input data; Based on the task type, extract corresponding key fields from the original input data; wherein, the key fields include at least one of the following: question text, reference answer text, image path, and video frame path set; The key fields are filled into a predefined standardized template to form the structured input sample.

3. The evaluation method for a modular large language model according to claim 1, characterized in that, Based on the task type, the corresponding multimodal large language model is dynamically matched and scheduled from the registered model services, specifically including: Query the pre-configured task-model mapping table to obtain one or more preferred model identifiers for the task type corresponding to the original input data; The model services corresponding to the preferred model identifiers are invoked in order of priority. If the highest priority model service call fails, the alternative model service will be automatically called as a fallback.

4. The evaluation method for a modular large language model according to claim 1, characterized in that, The original output content undergoes structural extraction, specifically including: Based on the task type, a corresponding structure extraction strategy is selected from a predefined rule base; Based on the selected structure extraction strategy, the core semantic content is extracted from the original output content, and preset template statements and redundant information are excluded.

5. The evaluation method for a modular large language model according to claim 4, characterized in that, The original output content undergoes semantic normalization processing, specifically including: Normalize the text in the original output content for capitalization and punctuation. Standardize the physical units mentioned in the text; The keywords in the original output content are extracted using an unsupervised keyword extraction algorithm, and the importance score of each sentence in the original output content is calculated based on the TextRank algorithm. Redundant sentences are removed based on sentence similarity thresholds, and after removing redundant sentences, a preset number of sentences are retained based on the importance score of each sentence.

6. The evaluation method for a modular large language model according to claim 1, characterized in that, Based on the task type, the corresponding automatic evaluation algorithm is invoked to calculate the normalized answer content, specifically including: If the task type is a text generation task, calculate the text similarity index between the normalized answer content and the reference answer based on n-gram matching; If the task type involves an image modality, the visual-language joint embedding model calculates the semantic similarity between the normalized answer content and the reference image; If the task type involves a video modality, the calculation is performed based on the semantic aggregation results of the video frames.

7. The evaluation method for a modular large language model according to claim 1, characterized in that, Constructing prompt words containing the standardized answer content and evaluation instructions for scoring using a pre-defined auxiliary scoring large language model, specifically including: Construct an assessment instruction template; wherein the assessment instruction template includes multiple subjective dimensions to be assessed, raw input data, standardized answer content, and optional reference answers; The evaluation instruction template is sent to the auxiliary scoring language model, and the auxiliary scoring language model is requested to return the scoring results of each dimension in a structured data format; Perform format validation on the returned scoring results.

8. The evaluation method for a modular large language model according to claim 1, characterized in that, Based on the automatic indicator scores and the subjective indicator scores, a structured evaluation report containing visualization content is generated, specifically including: The automatic indicator scores and the subjective indicator scores are combined according to predefined weights to obtain a comprehensive evaluation score; Based on the task type and the comprehensive evaluation score, generate a visual chart that includes indicator comparison, semantic heatmap and weakness analysis; The visualization charts, the standardized answer content, and the original input data are combined according to a preset template to output a structured evaluation report in at least one format.

9. An evaluation device for a modular large language model, characterized in that, The device includes: At least one processor; And, a memory communicatively connected to the at least one processor; The memory stores instructions that can be executed by the at least one processor, which, when executed by the at least one processor, enables the at least one processor to perform an evaluation method for a modular large language model as described in any one of claims 1-8.

10. A non-volatile computer storage medium storing computer-executable instructions, characterized in that, When the computer-executable instructions are executed, they implement the evaluation method for a modular large language model as described in any one of claims 1-8.

Images