A method and device for evaluating a multi-modal model, a storage medium and a product

Abstract

The application discloses a kind of multi-modal model's evaluation method, device, equipment, storage medium and product, it is related to computer technical field, the method includes: obtaining the evaluation request input by user, request includes the business scene to be evaluated, multi-modal input type, index threshold and compliance condition, and evaluation demand;The business scene to be evaluated, multi-modal input type, index threshold and compliance condition are processed, obtain SCL description of safe arrangement language;SCL description is verified, and verification result is obtained;If the verification result represents that verification passes, SCL description is converted into abstract syntax tree AST description;According to evaluation demand and AST description, generate evaluation directed acyclic graph DAG, risk policy set and resource scheduling plan;According to DAG, risk policy set and resource scheduling plan, use sample data to evaluate multi-modal model, and obtain evaluation result.The method can improve the evaluation efficiency of multi-modal model.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of computers, and in particular to a multi-modal model evaluation method and device, equipment, storage medium and product. BACKGROUND

[0002] With the rapid development of artificial intelligence technology, multi-modal artificial intelligence models (covering text, images, speech, video and cross-modal fusion models) are increasingly widely used in medical, financial, content security and many other fields.

[0003] The performance evaluation of existing multi-modal models usually requires evaluation personnel to manually log in to multiple systems such as workflow platforms, policy databases, and monitoring databases, collect Directed Acyclic Graph (DAG) execution logs, detection results, performance indicators and other data one by one, and then generate a summary evaluation report through manual sorting, calculation and analysis.

[0004] The above traditional solution has the problem of poor evaluation efficiency, which is difficult to meet the large-scale and high-frequency evaluation needs of multi-modal models. SUMMARY

[0005] The present application provides a multi-modal model evaluation method, device, equipment, storage medium and product, which can improve the evaluation efficiency of multi-modal models.

[0006] To achieve the above purpose, the present application adopts the following technical solutions:

[0007] In a first aspect, the present application provides a multi-modal model evaluation method, comprising:

[0008] Obtaining an evaluation request input by a user, the evaluation request including a business scenario to be evaluated, a multi-modal input type, an index threshold and a compliance condition, and an evaluation requirement;

[0009] Standardizing the business scenario to be evaluated, the multi-modal input type, the index threshold and the compliance condition to obtain a Security Orchestration Language (SCL) description;

[0010] Verifying the SCL description to obtain a verification result;

[0011] If the verification result indicates that the verification is passed, converting the SCL description into an Abstract Syntax Tree (AST) description;

[0012] Generating an evaluation Directed Acyclic Graph (DAG), a risk policy set and a resource scheduling plan according to the evaluation requirement and the AST description;

[0013] According to the DAG, the risk policy set and the resource scheduling plan, the multi-modal model is evaluated by using sample data, and an evaluation result is obtained.

[0014] Optionally, the verifying the SCL description comprises:

[0015] The SCL description is verified from the dimensions of multi-modal type, industry consistency and policy operator matching degree.

[0016] Optionally, the method further comprises:

[0017] If the verification result represents that the verification fails, a prompt information is generated, and the prompt information comprises a verification dimension that fails the verification.

[0018] Optionally, the method further comprises:

[0019] Obtaining environment information corresponding to the evaluation request;

[0020] Based on the evaluation result, an evaluation report carrying the environment information is generated.

[0021] Optionally, the verifying the SCL description from the dimensions of multi-modal type, industry consistency and policy operator matching degree comprises:

[0022] Obtaining a first verification rule corresponding to the multi-modal type, a second verification rule corresponding to the industry consistency and a third verification rule corresponding to the policy operator matching degree;

[0023] The SCL description is verified by using the first verification rule, the second verification rule and the third verification rule.

[0024] In a second aspect, the application provides a multi-modal model evaluation device, comprising:

[0025] An obtaining module is configured to obtain an evaluation request input by a user, wherein the evaluation request comprises a business scenario to be evaluated, a multi-modal input type, an index threshold value and a compliance condition, and an evaluation requirement;

[0026] A processing module is configured to perform standardization processing on the business scenario to be evaluated, the multi-modal input type, the index threshold value and the compliance condition, to obtain a security orchestration language (SCL) description; verify the SCL description to obtain a verification result; if the verification result represents that the verification passes, convert the SCL description into an abstract syntax tree (AST) description; and generate an evaluation directed acyclic graph (DAG), a risk policy set and a resource scheduling plan according to the evaluation requirement and the AST description.

[0027] An evaluation module is configured to evaluate the multi-modal model using sample data according to the DAG, the risk policy set, and the resource scheduling plan, and obtain an evaluation result.

[0028] In a third aspect, the present application provides a computing device, comprising a memory and a processor;

[0029] One or more computer programs are stored in the memory, and the one or more computer programs comprise instructions; when the instructions are executed by the processor, the computing device performs the method according to any one of the first aspect.

[0030] In a fourth aspect, the present application provides a computer readable storage medium for storing a computer program, wherein the computer program is used to perform the method according to any one of the first aspect.

[0031] According to the above technical solution, the present application has at least the following beneficial effects:

[0032] In the present application, the multi-modal model evaluation method solves the main problem of poor evaluation efficiency of the traditional scheme by standardizing SCL description, multi-dimensional verification, AST conversion, and automatically generating evaluation DAG, risk policy set, and resource scheduling plan.

[0033] On the one hand, it unifies the to-be-evaluated business scenarios, multi-modal input types, index thresholds, and compliance conditions into standardized SCL descriptions, avoiding the cumbersome operations of multi-system scattered configuration in the traditional scheme, and reducing invalid execution and manual correction costs in the subsequent evaluation process from the source through three verifications of multi-modal types, industry consistency, and strategy operator matching degree; on the other hand, based on AST description, it automatically generates executable evaluation processes and resource configurations, without the need for manual construction of DAG, scattered collection of multi-source data, and arrangement and analysis, realizing full-process automation of the closed loop from evaluation request input to evaluation result output.

[0034] In addition, the evaluation report is automatically associated with environmental information, further saving the time for manual tracing of environmental configuration, greatly improving the execution efficiency in the large-scale and high-frequency evaluation scenarios of multi-modal models, and effectively adapting to the demand for rapid iteration and evaluation of models under the background of wide application of artificial intelligence technology.

[0035] It should be understood that the descriptions of technical features, technical solutions, beneficial effects, or similar language in this application do not imply that all features and advantages can be achieved in any single embodiment. Rather, it is understood that the description of a feature or beneficial effect means that a specific technical feature, technical solution, or beneficial effect is included in at least one embodiment. Therefore, the descriptions of technical features, technical solutions, or beneficial effects in this specification do not necessarily refer to the same embodiment. Furthermore, the technical features, technical solutions, and beneficial effects described in this embodiment can be combined in any suitable manner. Those skilled in the art will understand that embodiments can be implemented without one or more specific technical features, technical solutions, or beneficial effects of a particular embodiment. In other embodiments, additional technical features and beneficial effects may be identified in specific embodiments that do not embody all embodiments. Attached Figure Description

[0036] Figure 1 A flowchart illustrating a method for evaluating a multimodal model provided in this application embodiment;

[0037] Figure 2 A schematic diagram of a multimodal model evaluation device provided in an embodiment of this application;

[0038] Figure 3 This is a schematic diagram of a computing device provided in an embodiment of this application. Detailed Implementation

[0039] The terms "first," "second," and "third," etc., used in this application specification and accompanying drawings are used to distinguish different objects, not to limit a specific order.

[0040] In the embodiments of this application, the terms "exemplary" or "for example" are used to indicate that something is an example, illustration, or description. Any embodiment or design that is described as "exemplary" or "for example" in the embodiments of this application should not be construed as being more preferred or advantageous than other embodiments or design. Specifically, the use of the terms "exemplary" or "for example" is intended to present the relevant concepts in a specific manner.

[0041] To ensure clarity and conciseness in the description of the following embodiments, a brief introduction to the related technologies is given first:

[0042] Multimodal models are artificial intelligence models that can process various types of data such as text, images, voice, and video, and achieve cross-modal information fusion and reasoning. They are widely used in fields such as medical diagnosis, financial risk control, and content security review.

[0043] With the increasing application of multimodal models in fields such as healthcare, finance, and content security, efficient and comprehensive performance evaluation and compliance verification have become critical requirements. However, existing technical solutions suffer from significant technical problems, such as poor evaluation efficiency, making them unsuitable for large-scale, high-frequency evaluation scenarios. In traditional solutions, evaluators need to manually log into multiple systems to collect data such as DAG execution logs, test results, and performance indicators, and then manually organize and analyze the data to generate summary reports. This process involves numerous manual interventions, a large amount of repetitive work, and is prone to extending the evaluation cycle due to human error, failing to meet the evaluation needs of rapid iteration of multimodal models.

[0044] The main reason for this technical problem is that traditional solutions have not established a complete chain for the standardization, verification, and automated execution of assessment requirements. On the one hand, assessment elements such as business scenarios, multimodal input types, indicator thresholds, and compliance conditions lack unified description standards and need to be configured separately in different systems, which is prone to logical conflicts and cumbersome configuration processes. On the other hand, there is a lack of an automated conversion mechanism from user assessment requirements to executable processes. It is necessary to rely on manual construction of assessment DAGs and adjustment of resource parameters. At the same time, the rationality of assessment requirements is not systematically verified in advance, which may lead to invalid execution due to problems such as multimodal type mismatch and incompatibility between compliance conditions and execution operators, further reducing assessment efficiency.

[0045] In view of this, embodiments of this application provide a method for evaluating multimodal models, which can be executed by a processing device. The processing device can be a terminal or a server. Terminals include, but are not limited to, smartphones, tablets, laptops, personal digital assistants, or smart wearable devices. The server can be a cloud server, such as a central server in a central cloud computing cluster or an edge server in an edge cloud computing cluster. Alternatively, the server can be a server in a local data center. A local data center refers to a data center directly controlled by the user.

[0046] To address the issue of poor evaluation efficiency in existing multimodal model evaluation solutions, this application designs a unified Security Orchestration Language (SCL) to transform dispersed business scenarios, multimodal input types, indicator thresholds, and compliance conditions into standardized descriptions. These descriptions are then processed through multi-dimensional verification and Abstract Syntax Tree (AST) conversion, bridging user needs with automated execution. Based on the AST descriptions, a Directed Acyclic Graph (DAG), risk strategy set, and resource scheduling plan are automatically generated, enabling integrated and automated implementation of the evaluation process, compliance verification, and resource allocation. Simultaneously, reports are generated by linking evaluation environment information. This addresses the pain points of inconsistent descriptions, manual processes, and lack of automated conversion and pre-verification in traditional solutions, significantly improving the efficiency and reliability of large-scale, high-frequency multimodal model evaluation.

[0047] To make the technical solution of this application clearer and easier to understand, the following describes a method for evaluating a multimodal model provided by an embodiment of this application, in conjunction with the accompanying drawings. Figure 1 As shown, this figure is a flowchart of a multimodal model evaluation method provided in an embodiment of this application. The method includes:

[0048] S201. The processing device obtains the user input evaluation request. The evaluation request includes the business scenario to be evaluated, multimodal input type, indicator threshold and compliance conditions, and evaluation requirements.

[0049] An evaluation request is a set of instructions initiated by the user to the processing device, containing all the requirements related to multimodal model evaluation. It is the input that triggers subsequent evaluation processes such as standardization conversion, verification, and automated execution.

[0050] The business scenario to be evaluated refers to the actual application scenario of the multimodal model, which clarifies the industry or business background of the evaluation, such as medical image diagnosis, financial risk control audit, content security testing, etc., in order to adapt to scenario-based evaluation rules and indicators.

[0051] Multimodal input types refer to the types of data that the model to be evaluated can process, including single modal data such as text, images, speech, and video, or cross-modal data combinations such as text and images, or speech and video. These determine the types of data sources and data processing methods required during the evaluation process.

[0052] Metric thresholds are quantitative standards set by users to measure whether a model passes evaluation. They include performance metrics, such as accuracy ≥ 0.9, P95 latency < 300ms, throughput ≥ 10QPS, and cost metrics, such as cost per evaluation < $0.05. These serve as the basis for determining subsequent evaluation results.

[0053] Compliance conditions are the compliance and security rules proposed by users that must be followed during the model evaluation process, such as not disclosing privacy information, not outputting illegal or non-compliant content, and avoiding algorithmic bias. They are key inputs for generating risk strategy sets and ensuring evaluation compliance.

[0054] Evaluation requirements are a collective term for the main objectives and additional requirements of the user for this evaluation. In addition to the above-mentioned business scenarios, input types, indicator thresholds, and compliance conditions, they may also include evaluation report formats, result traceability requirements, deployment environment restrictions, etc., which are used to guide the execution of the entire evaluation process.

[0055] The processing device first receives the evaluation instruction, or evaluation request, initiated by the user. This evaluation instruction must fully include five categories of key information: clearly defining the actual application scenario of the model, i.e., the business scenario to be evaluated; clearly defining the data types that the model can process, i.e., multimodal input types; clearly defining the quantitative standards for measuring model performance, i.e., indicator thresholds; clearly defining the compliance and security rules that must be followed during the evaluation process, i.e., compliance conditions; and clearly defining the user's main objectives and additional requirements for this evaluation, i.e., evaluation needs.

[0056] This information forms the basis for subsequent evaluation processes such as standardized conversion, multi-dimensional verification, and automated generation of evaluation DAGs, ensuring that the processing equipment accurately understands user needs and can then conduct targeted multimodal model evaluations.

[0057] S202. The processing device standardizes the business scenarios, multimodal input types, indicator thresholds, and compliance conditions to be evaluated, and obtains the Security Orchestration Language (SCL) description.

[0058] Standardization processing refers to the process by which processing equipment organizes, transforms, and standardizes scattered evaluation-related information according to preset unified rules, formats, and semantic specifications. The purpose is to eliminate differences in information expression and form a unified format that can be recognized and parsed by machines.

[0059] Security Orchestration Language (SCL) is a standardized description language specifically designed for this application. It is used to uniformly represent the core elements of multimodal model evaluation and can transform information such as business scenarios, data types, indicator requirements, and compliance rules into structured and logically consistent language expressions. It is a key carrier connecting user needs with automated evaluation execution.

[0060] After receiving a user's evaluation request, the processing device initiates a pre-set unified rules and format adaptation process for four key information categories: the business scenario to be evaluated, the multimodal input type, the indicator threshold, and the compliance conditions. Through standardized organization and semantic conversion, the potentially scattered and unstructured information is transformed into a logically consistent and formatted Security Orchestration Language (SCL) description.

[0061] This step eliminates discrepancies in the representation of different information, providing a unified input basis for subsequent multi-dimensional verification, abstract syntax tree (AST) transformation, and automated generation of evaluation DAGs, ensuring the automation and accuracy of the entire evaluation process.

[0062] S203. The processing device verifies the SCL description and obtains the verification result.

[0063] Verification refers to the process by which processing equipment systematically checks the completeness, rationality, compatibility, and executability of SCL descriptions according to a preset rule system. The purpose is to verify whether the SCL description accurately maps user requirements and whether there are any logical conflicts, type errors, or other issues, thus laying the foundation for subsequent automated processes.

[0064] The verification result is a structured feedback message output after the verification is completed. It is divided into two categories: verification passed and verification failed. When it passes, only a confirmation mark is returned. When it fails, the dimension that failed (such as multimodal type mismatch, policy operator incompatibility) and the specific reason must be clearly marked so that the user can correct the SCL description.

[0065] After obtaining a standardized SCL description, the processing device will automatically call the preset verification rule system, including multimodal type compatibility rules, industry terminology consistency rules, strategy and operator matching rules, etc., to conduct a comprehensive check on the business scenario adaptability, multimodal input type rationality, indicator threshold validity, compliance condition executability, etc. in the SCL description.

[0066] Specifically, the processing device verifies the SCL description from the dimensions of multimodal type, industry consistency, and policy operator matching degree.

[0067] The multimodal type verification dimension focuses on the compatibility check between the multimodal input types in the SCL description and the evaluation process and operator capabilities, verifying whether the input modality matches the processing logic, such as whether the text modality is compatible with image processing operators.

[0068] The industry consistency verification dimension is based on the standard terminology, business rules and regulatory requirements of a specific industry. It checks whether the scenario descriptions, indicator definitions and compliance conditions in the SCL description conform to industry standards. For example, whether the terminology in diagnostic reports in medical scenarios is standardized.

[0069] The strategy operator matching verification dimension verifies whether the compliance conditions (policies) defined in the SCL description are compatible with the execution components (operators) already deployed in the system, ensuring that each compliance rule has a corresponding technical component to support its execution. For example, the PII desensitization policy needs to match the PII detection operator.

[0070] When validating SCL descriptions, the processing equipment conducts a systematic check across three dimensions. First, in the multimodal type dimension, it verifies the compatibility of the input modality type (such as text and images) with the processing logic and operator capabilities of subsequent evaluation tasks, avoiding type conflicts such as using text operators to process image data. Second, in the industry consistency dimension, it compares the SCL description with industry standards for corresponding business scenarios, such as risk control rules in the financial industry and terminology standards in the medical industry, verifying whether the scenario descriptions and indicator definitions comply with industry practices and regulatory requirements. Third, in the strategy operator matching dimension, it checks each compliance condition in the SCL description, such as privacy protection and content security rules, confirming that the system has deployed the corresponding execution operators, such as PII detection operators and sensitive word blocking operators, ensuring that compliance rules can be effectively implemented.

[0071] The specific verification steps are as follows:

[0072] The processing device obtains the first verification rule corresponding to the multimodal type, the second verification rule corresponding to the industry consistency, and the third verification rule corresponding to the strategy operator matching degree.

[0073] The first verification rule is a standardized set of checking rules specifically designed for multimodal type compatibility. It includes modal type matching rules, cross-modal data flow rules, etc., and is used to verify the adaptability of multimodal input types in the SCL description to the evaluation process and operator capabilities.

[0074] The second verification rule is a set of rules based on industry standards, business practices and regulatory requirements. It covers industry terminology standards, scope of scenario-based indicators, compliance baseline requirements, etc., and is used to ensure that the SCL description is consistent with the industry attributes of the corresponding business scenario.

[0075] The third set of verification rules focuses on the compatibility between compliance policies and execution operators. It includes rules for mapping policy types to operator functions, rules for verifying operator deployment status, etc., to ensure that each compliance condition has corresponding technical component support.

[0076] Rule acquisition involves the processing device obtaining three types of validation rules through methods such as calling the built-in rule library, pulling industry rule interfaces, or importing user-defined rules. The rules are stored in structured formats, such as JSON and YAML, and support automatic parsing and execution.

[0077] The processing equipment will acquire three types of targeted verification rules in advance. The first verification rule corresponds to the multimodal type verification dimension, which clarifies which modal types can be adapted to which processing operators, and the judgment criteria for how cross-modal data should be transferred legally. The second verification rule corresponds to the industry consistency verification dimension, which includes a list of standard terms for specific industries, reasonable ranges of indicators, and compliance and regulatory red lines, such as the risk scoring indicator range for the financial industry and the definition of sensitive data for the medical industry. The third verification rule corresponds to the strategy operator matching degree verification dimension, which aims to establish a mapping relationship table between strategies and operators (such as the PII desensitization strategy corresponding to the PII detection operator and field replacement operator) and check the operator deployment status.

[0078] These rules serve as the basis for subsequent verification of the processing equipment, ensuring that the verification process is systematic and the results are accurate and reliable.

[0079] The processing device uses the first verification rule, the second verification rule, and the third verification rule to verify the SCL description.

[0080] After acquiring the three types of verification rules, the processing device will start an automated verification process, applying the first, second, and third verification rules to the corresponding dimensions described by the SCL to conduct targeted checks.

[0081] Specifically, the first verification rule, namely the multimodal type compatibility rule, is used to verify the compatibility between the input modality type in the SCL description and the processing operators and evaluation process. For example, it verifies whether text and image modalities can be processed normally by subsequent operators. The second verification rule, namely the industry consistency rule, is used to verify whether the business terms, indicator ranges, and compliance requirements in the SCL description conform to the corresponding industry standards. For example, the reasonable range of risk indicators in financial scenarios and the definition of sensitive data in medical scenarios. The third verification rule, namely the strategy operator matching degree rule, is used to verify whether each compliance strategy in the SCL description has a corresponding execution operator to support it. For example, whether the PII desensitization strategy can match the PII detection operator and desensitization operator already deployed in the system.

[0082] The three types of rules operate in parallel and comprehensively cover the key dimensions of the SCL description. The processing device will automatically compare the SCL description with the rule requirements and output the result of whether the verification passes or fails, ensuring that the verification process is efficient, accurate and complete.

[0083] S204. If the verification result indicates that the verification is successful, then the SCL description is converted into an Abstract Syntax Tree (AST) description.

[0084] Abstract Syntax Tree (AST) descriptions are structured tree-like data structures generated from standardized SCL descriptions through syntactic parsing and semantic extraction. They clearly present the logical structure of assessment requirements through a hierarchical relationship of root nodes, child nodes, and leaf nodes. For example, the business scenario is the root node, multimodal inputs, indicator thresholds, and compliance conditions are child nodes, and specific parameters are leaf nodes. ASTs serve as an intermediate carrier that machines can directly parse and use for subsequent automated generation of assessment DAGs and risk strategy sets.

[0085] Transformation refers to the process by which processing devices use built-in parsers, such as ANTLR and Xtext, to perform lexical analysis, syntactic analysis, and semantic mapping on validated SCL descriptions, stripping away natural language style expressions, extracting logic and data associations, and ultimately constructing a hierarchical and logically rigorous AST structure.

[0086] When the verification result output by the processing device is that the verification is successful, that is, the SCL description meets the rule requirements in dimensions such as multimodal type compatibility, industry consistency, and policy operator matching degree, the syntax parsing and structure conversion process will be initiated.

[0087] The processing device first performs lexical analysis on the SCL description through a built-in parser to identify basic syntactic units such as keywords, parameter values, and logical symbols. Then, it performs syntactic analysis to verify whether the combination of syntactic units conforms to the SCL language specification. Finally, it transforms information such as business scenarios, multimodal input types, indicator thresholds, and compliance conditions in the SCL into an AST description organized in a tree structure through semantic mapping.

[0088] For example, in the financial risk control scenario in SCL, text and image inputs, accuracy > 0.9, and PII anonymization compliance conditions will be parsed as follows: the root node is financial risk control scenario, and the sub-nodes are multimodal inputs, indicator thresholds, and compliance conditions. The leaf nodes of multimodal inputs are text and images, the leaf nodes of indicator thresholds are accuracy, 0.9, and >, and the leaf node of compliance conditions is PII anonymization. The hierarchical relationship between the nodes is clearly defined through logical association.

[0089] This AST description eliminates redundant expressions in the SCL description, retaining only logical and data associations, providing accurate and structured machine-readable input for the subsequent automatic generation of executable files such as evaluation DAGs and risk strategy sets.

[0090] If the verification result indicates that the verification failed, a prompt message will be generated, which will include the verification dimension that failed the verification.

[0091] The prompt message refers to the structured feedback data generated by the processing device when the verification fails. It includes the failed verification dimension, specific problem description, cause analysis and correction suggestions. It is output in both machine-readable and human-readable form to facilitate users to quickly locate and correct the problem.

[0092] After the processing device completes the verification of the SCL description through three types of verification rules, if the verification result fails, that is, at least one verification dimension does not meet the rule requirements, a targeted prompt message will be automatically generated.

[0093] One of the main contents of this prompt message is to clearly mark the specific dimensions that failed the verification, such as the multimodal type dimension, industry consistency dimension, and strategy operator matching degree dimension. At the same time, it combines the quantitative calculation results recorded during the verification process, such as modality-operator matching degree score, industry terminology similarity, and strategy-operator function adaptability, and supplements the specific problem description, such as incompatibility between text and image modalities and text processing operators, accuracy threshold of 0.7 exceeding the reasonable range of 0.8~0.98 in the financial industry, and PII desensitization strategy with existing operator function adaptability of only 0.5, which does not reach the threshold of 0.8, as well as the corresponding cause analysis and correction suggestions, such as replacing with operators that support cross-modal processing, adjusting the accuracy threshold to above 0.8, and deploying dedicated operators that include PII detection and field replacement functions.

[0094] This prompt allows users to quickly locate the main issues and their corresponding dimensions in the SCL description without having to check each one individually, greatly improving the efficiency of problem correction and ensuring that the subsequent SCL description can successfully pass the verification and enter the AST conversion stage.

[0095] S205. The processing equipment generates the evaluation directed acyclic graph (DAG), risk strategy set, and resource scheduling plan based on the evaluation requirements and AST description.

[0096] The evaluation directed acyclic graph (DAG) is a loop-free execution flow model constructed from nodes and directed edges. It is used to define the execution order, data flow, and dependency constraints of evaluation tasks and serves as the execution carrier for automated evaluation.

[0097] The risk strategy set is a set of structured rules generated based on the compliance conditions and assessment requirements described in the AST. It includes a triplet logic of triggering conditions, execution actions, and judgment criteria, which is used to automatically implement compliance requirements and avoid security risks during the assessment process.

[0098] The resource scheduling plan is a hardware / software resource allocation scheme generated by combining the indicator thresholds (such as latency and throughput) in the AST description with the complexity of the evaluation task. It includes parameters such as resource allocation ratio, task concurrency, batch size, and model routing strategy. The goal is to optimize resource utilization efficiency while meeting the indicator requirements.

[0099] Specifically, the processing device generates an evaluation DAG that defines the execution order and data flow of evaluation tasks based on the hierarchical logic and task dependencies described in the AST description; it generates a risk strategy set containing triggering conditions, execution actions, and judgment criteria based on the compliance conditions in the AST description and the security requirements in the evaluation requirements, to ensure that the evaluation process is compliant and controllable; and it generates a scheduling plan that optimizes resource allocation based on the quantitative indicator thresholds (such as latency ≤300ms) in the AST description and the complexity of the evaluation tasks, such as sample size and model parameter count, to clarify the configuration parameters such as computing power, memory, and concurrency.

[0100] The three types of documents work together to form a complete execution plan for automated assessment. The DAG defines what to do and in what order, the risk strategy set defines how to prevent compliance risks, and the resource scheduling plan defines how to allocate resources. Together, they support the efficient, compliant, and stable execution of the assessment process, achieving seamless integration from user needs to automated execution.

[0101] S206. The processing equipment evaluates the multimodal model using sample data based on the DAG, risk strategy set, and resource scheduling plan, and obtains the evaluation results.

[0102] Sample data refers to datasets used to verify the performance and compliance of multimodal models. It must match the multimodal input types in the evaluation request, such as text / image / speech / video, as well as the business scenario. It includes labeled and unlabeled data. Labeled data is used to calculate metrics such as accuracy and recall, while unlabeled data is used for compliance risk detection. Its sample size and distribution characteristics directly affect the reliability of the evaluation results.

[0103] The processing equipment uses evaluation DAG, risk strategy set and resource scheduling plan as the basis for execution, and takes sample data matching business scenarios and modal types as input to carry out full-process automated evaluation of multimodal models.

[0104] Specifically, firstly, according to the task order and dependencies defined in the evaluation DAG, tasks such as data preprocessing, model inference, indicator calculation, and compliance verification are executed sequentially. During execution, the rule logic of the risk strategy set is embedded to automatically detect compliance risks, i.e., sensitive information leakage, in sample data processing and model output, and to execute corresponding control actions, i.e., desensitization and interception. At the same time, hardware computing power is dynamically allocated and task concurrency and batch processing size are adjusted according to the resource scheduling plan to ensure that the evaluation process proceeds efficiently while meeting the performance indicator thresholds, i.e., latency requirements.

[0105] Finally, the processing equipment integrates information such as task execution logs, indicator calculation results, and compliance verification records to generate evaluation results that include performance compliance status, compliance status, and risk events, providing a direct basis for subsequent report generation and model optimization.

[0106] The method also includes generating an assessment report containing environmental information, with the following specific steps:

[0107] The processing equipment acquires the environmental information corresponding to the evaluation request; based on the evaluation results, it generates an evaluation report carrying the environmental information.

[0108] Environmental information refers to the underlying configuration and runtime context data related to this evaluation request, including hardware environment such as CPU / GPU model, memory size, video memory capacity, software environment such as operating system version, model framework version, dependent library version, dataset information such as dataset name, version, sample size and distribution, and deployment environment such as local server, cloud computing node, containerized cluster, etc. It serves as the basis for tracing the validity and reproducibility of the evaluation results.

[0109] The assessment report is a structured output document that integrates key information from the entire assessment process. It includes modules such as a summary of assessment requirements, details of assessment execution, quantitative indicator results, compliance verification conclusions, and environmental information filing. It is both machine-readable and human-readable, and is used to support model optimization decisions, compliance audits, and result traceability.

[0110] On the one hand, after receiving the evaluation request, the processing device will simultaneously collect the environmental information corresponding to this evaluation, and comprehensively record the underlying context data such as hardware configuration, software version, dataset attributes, and deployment architecture to ensure the traceability of the evaluation process. On the other hand, after completing the automated evaluation based on DAG, risk strategy set and resource scheduling plan, the processing device will associate and integrate the generated evaluation results, including indicator compliance status, compliance execution logs, risk event records, etc., with the environmental information collected in the early stage to generate a complete evaluation report.

[0111] This report not only clearly presents the model's performance and compliance status in the current environment, but also provides key support for subsequent reproduction of test results, analysis of the impact of environmental factors on test conclusions (i.e., the impact of hardware configuration on latency), and optimization of model deployment schemes by linking environmental information.

[0112] Based on the above description, this application has the following beneficial effects:

[0113] In this application, the multimodal model evaluation method solves the main problem of poor evaluation efficiency of traditional solutions by standardizing SCL description, multi-dimensional verification, AST transformation and automatic generation of evaluation DAG, risk strategy set and resource scheduling plan.

[0114] On the one hand, it unifies the business scenarios to be evaluated, multimodal input types, indicator thresholds, and compliance conditions into standardized SCL descriptions, avoiding the cumbersome operation of multi-system decentralized configuration in traditional solutions. At the same time, through triple verification of multimodal types, industry consistency, and policy operator matching degree, it reduces the cost of invalid execution and manual correction in the subsequent evaluation process from the source. On the other hand, it automatically generates directly executable evaluation processes and resource configurations based on AST descriptions, eliminating the need for manual DAG construction, decentralized collection of multi-source data, and data processing and analysis, thus achieving a fully automated closed loop from evaluation request input to evaluation result output.

[0115] In addition, the evaluation report automatically associates environmental information, further saving the time of manually tracing environmental configuration, and greatly improving the execution efficiency in multimodal model scaling and high-frequency evaluation scenarios, effectively adapting to the demand for rapid iterative evaluation of models in the context of the widespread application of artificial intelligence technology.

[0116] The above text combined Figure 1 The evaluation method for multimodal models provided in the embodiments of this application has been described in detail. The apparatus and equipment provided in the embodiments of this application will be described below with reference to the accompanying drawings.

[0117] like Figure 2 As shown in the figure, this is a schematic diagram of a multimodal model evaluation device provided in an embodiment of this application. The device includes:

[0118] The acquisition module 301 is used to acquire the assessment request input by the user. The assessment request includes the business scenario to be assessed, the multimodal input type, the indicator threshold and compliance conditions, and the assessment requirements.

[0119] The processing module 302 is used to standardize the business scenario, multimodal input type, indicator threshold and compliance conditions to be evaluated to obtain a Security Orchestration Language (SCL) description; to verify the SCL description and obtain the verification result; if the verification result indicates that the verification is passed, the SCL description is converted into an Abstract Syntax Tree (AST) description; and according to the evaluation requirements and the AST description, an evaluation Directed Acyclic Graph (DAG), a risk strategy set and a resource scheduling plan are generated.

[0120] The evaluation module 303 is used to evaluate the multimodal model using sample data based on the DAG, risk strategy set and resource scheduling plan, and obtain the evaluation results.

[0121] Optionally, the processing module 302 is specifically used to verify the SCL description from the dimensions of multimodal type, industry consistency, and policy operator matching degree.

[0122] Optionally, the processing module 302 is further configured to generate a prompt message if the verification result indicates that the verification failed, the prompt message including the verification dimension that failed the verification.

[0123] Optionally, the acquisition module 301 is also used to acquire environmental information corresponding to the assessment request;

[0124] The processing module 302 is also used to generate an evaluation report carrying the environmental information based on the evaluation results.

[0125] Optionally, the acquisition module 301 is specifically used to acquire the first verification rule corresponding to the multimodal type, the second verification rule corresponding to the industry consistency, and the third verification rule corresponding to the strategy operator matching degree.

[0126] The processing module 302 is specifically used to verify the SCL description using the first verification rule, the second verification rule, and the third verification rule.

[0127] The multimodal model evaluation apparatus according to the embodiments of this application can correspondingly execute the method described in the embodiments of this application, and the other operations and / or functions of each module / unit of the multimodal model evaluation apparatus are respectively for implementing Figure 1 For the sake of brevity, the corresponding processes of each method in the illustrated embodiments will not be described in detail here.

[0128] This application also provides a computing device. For example... Figure 3 As shown in the figure, this is a schematic diagram of a computing device provided in an embodiment of this application. The computing device 700 includes a bus 701, a processor 702, a communication interface 703, and a memory 704. The processor 702, the memory 704, and the communication interface 703 communicate with each other via the bus 701.

[0129] The 701 bus can be a Peripheral Component Interconnect (PCI) bus or an Extended Industry Standard Architecture (EISA) bus, etc. Buses can be categorized as address buses, data buses, control buses, etc. For ease of representation, Figure 3 The bus is represented by a single thick line, but this does not mean that there is only one bus or one type of bus.

[0130] The processor 702 can be any one or more of the following processors: central processing unit (CPU), graphics processing unit (GPU), microprocessor (MP), or digital signal processor (DSP).

[0131] The communication interface 703 is used for communication with external devices.

[0132] Memory 704 may include volatile memory, such as random access memory (RAM). Memory 704 may also include non-volatile memory, such as read-only memory (ROM), flash memory, hard disk drive (HDD), or solid state drive (SSD).

[0133] The memory 704 stores executable code, and the processor 702 executes the executable code to perform the aforementioned evaluation method for the multimodal model.

[0134] Specifically, in achieving Figure 2 In the case of the illustrated embodiment, and Figure 2 When the modules or units of the multimodal model evaluation device described in the embodiment are implemented by software, the following steps are performed: Figure 2 The software or program code required for the functions of each module / unit can be partially or wholly stored in memory 704. Processor 702 executes the program code corresponding to each unit stored in memory 704 to execute the aforementioned evaluation method for the multimodal model.

[0135] This application also provides a computer-readable storage medium. The computer-readable storage medium can be any available medium that a computing device can store, or a data storage device such as a data center containing one or more available media. The available medium can be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., solid-state drive). The computer-readable storage medium includes instructions that instruct the computing device to execute the aforementioned multimodal model evaluation method.

[0136] This application also provides a computer program product comprising one or more computer instructions. When the computer instructions are loaded and executed on a computing device, all or part of the processes or functions described in this application are generated.

[0137] The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another. For example, the computer instructions may be transmitted from one website, computer, or data center to another website, computer, or data center via wired (e.g., coaxial cable, fiber optic, digital subscriber line) or wireless (e.g., infrared, wireless, microwave, etc.) means.

[0138] When the computer program product is executed by a computer, the computer executes any of the aforementioned methods for evaluating the multimodal model. The computer program product can be a software installation package; when any of the aforementioned methods for evaluating the multimodal model is required, the computer program product can be downloaded and executed on the computer.

[0139] The descriptions of the processes or structures corresponding to the above figures each have their own emphasis. For parts of a process or structure that are not described in detail, please refer to the relevant descriptions of other processes or structures.

[0140] The above description is merely a specific embodiment of this application, but the scope of protection of this application is not limited thereto. Any changes or substitutions within the technical scope disclosed in this application should be covered within the scope of protection of this application.

Claims

1. A method for evaluating a multimodal model, characterized in that, The method includes: Obtain the user's input assessment request, which includes the business scenario to be assessed, multimodal input type, indicator threshold and compliance conditions, as well as assessment requirements. The assessment requirements are the user's main objectives and additional requirements for this assessment. The business scenarios, multimodal input types, indicator thresholds, and compliance conditions to be evaluated are standardized to obtain the Security Orchestration Language (SCL) description. The SCL description is verified to obtain the verification result; If the verification result indicates that the verification is successful, the SCL description is converted into an Abstract Syntax Tree (AST) description. Based on the hierarchical logic and task dependencies described by the AST, a directed acyclic graph (DAG) is generated to define the execution order and data flow of the evaluation tasks. Based on the compliance conditions in the AST description and the security requirements in the evaluation requirements, a risk strategy set containing triggering conditions, execution actions, and judgment criteria is generated. Based on the quantitative indicator thresholds and the complexity of the evaluation tasks in the AST description, a resource scheduling plan to optimize resource allocation is generated. Based on the DAG, risk strategy set, and resource scheduling plan, the multimodal model is evaluated using sample data to obtain evaluation results; Based on the DAG, risk strategy set, and resource scheduling plan, the multimodal model is evaluated using sample data, and the evaluation results include: According to the task order and dependencies defined in the evaluation DAG, data preprocessing, model inference, indicator calculation, and compliance verification tasks are executed sequentially. During execution, the rule logic of the risk strategy set is embedded to automatically detect compliance risks, i.e., sensitive information leakage, in sample data processing and model output, and execute corresponding control actions. At the same time, hardware computing power is dynamically allocated and task concurrency and batch processing size are adjusted according to the resource scheduling plan. Integrate task execution logs, indicator calculation results, and compliance verification records to generate evaluation results that include performance compliance status, compliance status, and risk events.

2. The method according to claim 1, characterized in that, The verification of the SCL description includes: The SCL description is validated from the dimensions of multimodal type, industry consistency, and policy operator matching degree.

3. The method according to claim 2, characterized in that, The method further includes: If the verification result indicates that the verification failed, a prompt message is generated, which includes the verification dimension that failed the verification.

4. The method according to claim 1, characterized in that, The method further includes: Obtain the environmental information corresponding to the assessment request; Based on the evaluation results, an evaluation report carrying the environmental information is generated.

5. The method according to claim 2, characterized in that, The verification of the SCL description from the dimensions of multimodal type, industry consistency, and policy operator matching degree includes: Obtain the first verification rule corresponding to the multimodal type, the second verification rule corresponding to the industry consistency, and the third verification rule corresponding to the strategy operator matching degree; The SCL description is verified using the first verification rule, the second verification rule, and the third verification rule.

6. A multimodal model evaluation device, characterized in that, The device includes: The acquisition module is used to acquire the assessment request input by the user. The assessment request includes the business scenario to be assessed, the multimodal input type, the indicator threshold and compliance conditions, as well as the assessment requirements. The assessment requirements are the user's main objectives and additional requirements for this assessment. The processing module is used to standardize the business scenarios, multimodal input types, indicator thresholds, and compliance conditions to be evaluated, obtaining a Security Orchestration Language (SCL) description; to validate the SCL description and obtain a validation result; if the validation result indicates that the validation is successful, the SCL description is converted into an Abstract Syntax Tree (AST) description; based on the hierarchical logic and task dependencies of the AST description, a Directed Acyclic Graph (DAG) defining the execution order and data flow of the evaluation tasks is generated; based on the compliance conditions in the AST description and the security requirements in the evaluation requirements, a risk strategy set containing triggering conditions, execution actions, and judgment criteria is generated; based on the quantitative indicator thresholds and the complexity of the evaluation tasks in the AST description, a resource scheduling plan for optimizing resource allocation is generated. The evaluation module is used to evaluate the multimodal model using sample data based on the DAG, risk strategy set, and resource scheduling plan, and obtain evaluation results. Specifically, it executes data preprocessing, model inference, indicator calculation, and compliance verification tasks sequentially according to the task order and dependencies defined in the evaluation DAG. During execution, it embeds the rule logic of the risk strategy set to automatically detect compliance risks, i.e., sensitive information leakage, in sample data processing and model output, and executes corresponding control actions. At the same time, it dynamically allocates hardware computing power and adjusts task concurrency and batch size according to the resource scheduling plan to ensure that the evaluation process proceeds while meeting performance indicator thresholds. It integrates task execution logs, indicator calculation results, and compliance verification records to generate evaluation results that include performance compliance status, compliance status, and risk event information.

7. The apparatus according to claim 6, characterized in that, The processing module is specifically used to verify the SCL description from the dimensions of multimodal type, industry consistency, and policy operator matching degree.

8. A computing device, characterized in that, Including memory and processor; The memory stores one or more computer programs, the one or more computer programs including instructions; when the instructions are executed by the processor, the computing device performs the method as described in any one of claims 1 to 5.

9. A computer-readable storage medium, characterized in that, The computer-readable storage medium is used to store a computer program for performing the method as described in any one of claims 1 to 5.

10. A computer program product, characterized in that, The computer program product includes one or more computer instructions, which, when executed by a computer, perform the method as described in any one of claims 1 to 5.

Images