System and method for security compliance assessment based on multi-modal large model

Abstract

The present application relates to security compliance assessment, in particular to a security compliance assessment system and method based on a multimodal large model, a function module cluster manages each function module in the system and supports cross-modal consistency detection for comparing the consistency of answers under different modes; a self-developed security system is responsible for the execution of the evaluation task, including adversarial test reinforcement, a multi-granularity data variation strategy is built-in to improve the evaluation robustness by simulating real attack scenarios, to realize end-to-end automation from data generation, tested model testing to risk assessment, support parameter configurable and result reproducible, and self-iteration and version management of the evaluation model, and the self-iteration mechanism is deeply combined with the compliance label system, which can continuously optimize the risk identification ability under the laws, ethics and industry specifications; the technical scheme provided by the present application can effectively overcome the limitation that the prior art can only evaluate under a single mode and a single rule.

Description

TECHNICAL FIELD

[0001] The present application relates to security compliance assessment, in particular to a security compliance assessment system and method based on a multi-modal large model. BACKGROUND

[0002] Currently, security compliance assessment systems mainly rely on manually designed test cases (such as sensitive word filtering, fixed question and answer templates, etc.) or rule-based keyword matching detection. For example, some systems use a pre-defined violation question library to test models in batches, or use traditional NLP classifiers to judge security compliance.

[0003] The above prior art solutions mainly have the following shortcomings:

[0004] 1) Insufficient coverage of evaluation

[0005] Problem: Existing systems rely too much on fixed data sets (such as pre-defined violation question libraries), making it difficult to cover new risks (such as induced answers specific to generative models).

[0006] Reason: Risk scenarios change dynamically, but the data set of traditional tools is updated with a lag, and lacks user-defined expansion capabilities.

[0007] 2) Lack of flexible data variation capabilities

[0008] Problem: Unable to perform diverse variation processing on the original data set (such as semantic rewriting, adversarial attack simulation, etc.), resulting in overfitting of evaluation results.

[0009] Reason: Existing systems usually treat data sets as static inputs and do not integrate data enhancement or adversarial testing modules.

[0010] 3) Limited model compatibility

[0011] Problem: Existing systems are often designed for specific APIs or model architectures (such as supporting only GPT series), making it difficult to adapt to different manufacturers' text generation models.

[0012] Reason: The input and output interfaces of different models differ greatly, and existing systems lack a unified protocol adaptation layer.

[0013] 4) Poor generalization of evaluation models

[0014] Problem: Risk assessment relies on rule engines or simple classification models (such as keyword matching), making it difficult to identify implicit ethical and legal risks (such as inducement of criminal rhetoric).

[0015] Reason: Traditional evaluation models do not incorporate multi-dimensional risk features (such as context coherence, intent concealment, etc.). SUMMARY

[0016] Technical problems solved

[0017] In view of the above-mentioned defects existing in the prior art, the present application provides a safety compliance evaluation system and method based on a multi-modal large model, which can realize safety compliance evaluation of intelligent model answers under multi-modal input such as text, image and audio, and overcome the limitation of the prior art that can only be evaluated under single mode and single rule through deep combination of cross-modal consistency detection and compliance label system.

[0018] Technical solutions

[0019] To achieve the above-mentioned purposes, the present application is realized by the following technical solutions:

[0020] The safety compliance evaluation system based on a multi-modal large model comprises:

[0021] A functional module cluster manages each functional module in the system and supports cross-modal consistency detection for comparing the consistency of answers under different modes.

[0022] A self-developed safety system is responsible for the execution of evaluation tasks, including adversarial test reinforcement, built-in multi-granularity data variation strategy, improved evaluation robustness through simulation of real attack scenarios, end-to-end automation from data generation, tested model testing to risk assessment, support for parameter configuration and result reproducibility, and self-iteration and version management of evaluation models, and the self-iteration mechanism is deeply combined with the compliance label system, which can continuously optimize the risk identification ability under the laws, ethics and industry standards.

[0023] Preferably, the functional module cluster is suitable for administrators and comprises user management, model management, tool management and data management.

[0024] In addition to supporting multiple interface types such as REST API, gRPC and SDK, the model management also supports protocol adaptation of cross-modal input and output to ensure unified access and management of multi-modal models including text, image and audio.

[0025] Preferably, the self-developed safety system is suitable for ordinary users and comprises:

[0026] Task execution is responsible for the whole process scheduling of evaluation tasks and automatically completes the "data processing-model interaction-result collection" process based on user configuration.

[0027] Evaluation report generation integrates multi-source evaluation results to generate a comprehensive evaluation report.

[0028] Model self-iteration realizes online iteration optimization of the evaluation model and improves the risk identification ability through closed-loop learning.

[0029] Preferably, the task execution specifically includes:

[0030] Task creation: the user selects the model under test, the original data set, the processing tool and the evaluation model, and sets the evaluation parameters;

[0031] Task distribution: allocate an independent running environment for the evaluation task to avoid multi-task interference;

[0032] Data injection: call the processing tool to perform variant processing on the original data set to generate a dynamic test set;

[0033] Model interaction: send the dynamic test set to the model under test through the protocol adaptation layer of the model management, and receive and store the answer results;

[0034] Progress monitoring: record the task progress in real time, and automatically retry in case of exception.

[0035] Preferably, the evaluation report generation specifically includes:

[0036] Evaluation index calculation: based on the evaluation results output by the evaluation model, calculate detailed evaluation indexes including risk detection rate, false positive rate and risk level distribution;

[0037] Multi-source comparison: compare the evaluation results of the evaluation model with the evaluation results output by the third-party model, and mark the difference items;

[0038] Evaluation report output: support PDF / Word format, including detailed evaluation indexes, typical case analysis and improvement suggestions;

[0039] The model self-iteration specifically includes:

[0040] Sample sorting: sort the "question-answer-risk label" triplets that have been evaluated into training set, validation set and test set;

[0041] Incremental training: train and optimize the evaluation model using the training set and the validation set, and improve the training efficiency by dynamically adjusting the learning rate;

[0042] Effect verification: verify the performance of the new evaluation model using the test set. If the performance improvement is greater than the preset threshold, update the evaluation model library, otherwise roll back to the historical version;

[0043] Version management: record the iteration history of the evaluation model, and support version traceability.

[0044] The safety compliance evaluation method based on the multi-modal large model is applied to the safety compliance evaluation system based on the multi-modal large model, and includes the following steps:

[0045] S1, task creation;

[0046] S2, data processing;

[0047] S3, the measured model is called;

[0048] S4, the evaluation model is called;

[0049] S5, cross-modal consistency detection: under multi-modal including text, image and audio, the consistency of the answer content of the measured model is compared, and the compliance label system is combined to judge.

[0050] Preferably, in S1, the task is created: the evaluation group leader creates the evaluation task, distributes it to the evaluators in real time, and monitors the specific evaluation process in real time, completing the closed-loop process of "task-execution-monitoring-result", including:

[0051] S11, the administrator logs in the system: the administrator logs in the system through the account password, and the system automatically verifies the identity information and the level of authority to ensure that the authorized administrator can enter the task management interface;

[0052] S12, evaluation task creation: the evaluation group leader fills in the basic information including task name, description and deadline in the task creation interface, selects the evaluators from the user list and assigns specific responsibilities, and clearly defines the evaluation requirements;

[0053] S13, evaluation task distribution: the system pushes the evaluation task details to the personal account of the designated evaluator in real time through the internal message pushing mechanism, and updates the task allocation state in the task board;

[0054] S14, independent running environment allocation: the system creates an isolated running environment for each evaluator to avoid data interference and resource preemption when multiple users operate, and to ensure the independence and stability of task execution;

[0055] S15, real-time monitoring progress: the system collects the task execution data of each evaluator in real time through the process monitoring tool, and displays it in the form of a visual chart on the evaluation group leader console.

[0056] Preferably, in S2, data processing: through the construction of "tool uploading-verification-registration-call" and "data uploading-variation-generation-storage" full-process mechanism, efficient generation and management of dynamic test set are realized, including:

[0057] S21, tool uploading: the user uploads the custom tool program through the standardized interface provided by the system;

[0058] S22, verify tool availability: the system deploys the tool package to an isolated test environment and automatically executes the preset test cases;

[0059] S23, tool library registration: tools that pass verification will be assigned a unique tool ID and associated with a function tag, stored in the tool library of the formal environment and opened to all users; tools that fail verification will provide specific failure reasons, and users can modify and re-upload after modification;

[0060] S24, upload raw data set: users upload structured raw data sets, the system supports batch import and automatically parses data fields, and provides repair suggestions for format errors;

[0061] S25, select tool chain: users check multiple tools in the tool library to form a tool chain, and adjust the execution order by dragging, the system provides tool compatibility prompts, and supports saving common tool chain templates for subsequent reuse;

[0062] S26, variant processing: the tool chain executes processing logic in order, and the anti-attack tool generates induced variants through synonym replacement, syntax reconstruction, and intent confusion, and records the processing results of each tool in real time;

[0063] S27, generate dynamic test set: aggregate all tool processing results, remove duplicate samples, and generate a dynamic test set containing original questions, variant questions, and corresponding tool IDs;

[0064] S28, data storage: the dynamic test set will be stored in the database, associated with the original data set ID, tool chain information, and generation time, while providing data export functions.

[0065] Preferably, the measured model in S3 is called: through the construction of a full-process mechanism of "model selection-connection adaptation-prompt generation-interactive storage", efficient calling and result management of different types of measured models are realized, including:

[0066] S31, select measured model: users select the type of measured model in the model list, API calling models need to fill in parameters such as interface address, access key, and request frequency limit; local models need to specify model file path and running environment;

[0067] S32, model connection: the protocol adaptation layer automatically generates requests that meet the target interface specifications for API calling models, and configures a timeout retry mechanism; for local models, automatically load model base files, initialize inference environment, and detect whether the model supports batch inference to improve efficiency;

[0068] S33, generate prompt words: the system generates standardized input text based on variant questions in the dynamic test set and pre-set prompt word templates;

[0069] S34, obtaining an answer result: the prompt word is sent to the measured model in an asynchronous calling mode, the request state is monitored in real time, task fragmentation processing is supported to avoid interface blocking, the answer result returned by the measured model is received, and irrelevant formats are automatically removed;

[0070] S35, storing an answer result: storing the answer result and information including the corresponding prompt word, measured model ID, request time and processing state.

[0071] Preferably, the evaluation model in S4 calls: by constructing a whole process mechanism of "multi-model calling-depth analysis-risk assessment-model iteration", the accurate evaluation of the answer result of the measured model and the continuous optimization of the evaluation system are realized, including:

[0072] S41, calling a multi-evaluation model: the system loads the selected model combination from the evaluation model library, the self-developed evaluation model is deployed on the local server to ensure the response speed, and the third-party model is called through the API interface;

[0073] S42, answer result analysis: multi-dimensional analysis is performed on the answer result of the measured model;

[0074] S43, comprehensive evaluation of risk level: according to the evaluation results output by the multi-evaluation model, an evaluation report is generated by integration;

[0075] S44, sample arrangement: the system automatically stores information including the questions in the dynamic test set, the answer result of the measured model, and the risk label corresponding to the evaluation result of the multi-evaluation model in the sample library according to time;

[0076] S45, evaluation model self-training: samples are extracted from the sample library in proportion, the sample size is expanded through data enhancement technology, and the evaluation model is trained and optimized, and the training loss is monitored in real time to avoid overfitting;

[0077] S46, model self-iteration: the new evaluation model needs to be verified for performance through the test set, a new version number is generated after verification, the old version model is replaced and iteration logs are recorded, and one-key rollback to the historical version is supported to deal with sudden situations.

[0078] (Three) beneficial effects

[0079] Compared with the prior art, the safety compliance evaluation system and method based on the multi-modal large model provided by the present application have the following beneficial effects:

[0080] 1) Improved dynamic risk coverage capability

[0081] The existing system relies on a static problem library, resulting in a low detection rate of new risks. The present application sets up a modular tool chain to support users in uploading custom data processing tools, realizes data expansion (such as new variants of "how to bypass bank risk control") and adversarial sample generation (integrates adversarial attack algorithms such as TextFooler, automatically generates induced questions such as changing "manufacturing bombs" to "household chemical pressure experiment scheme"), so that the dynamic expansion efficiency of the test set is improved by 300%, and the detection rate of new risks is improved from 42% of the existing system to 80%;

[0082] 2) Data variation and robustness enhancement

[0083] The existing system has fixed question and answer templates, resulting in the model learning "cheating skills" (such as detecting keywords and refusing to answer). The present application realizes semantic conservation variation (generates 200+ variations such as "stealing data" → "unauthorized access to information system resources" by dependency syntax analysis while keeping the core intent of the question unchanged) and context injection (inserting interfering context in the question, such as discussing network security first and then asking about cracking methods) in the self-developed security system, so that the model "cheating avoidance" behavior is reduced by 72%, and the stability of the evaluation results is improved by 58%;

[0084] 3) Breakthrough in full-stack model compatibility

[0085] The existing system has a special interface, which cannot evaluate heterogeneous models. The present application realizes the dual-channel design of model management, and the protocol adaptation layer uniformly encapsulates multiple interface types such as REST API, gRPC and SDK (such as handling the protobuf protocol of DeepSeek and the HTTP-JSON protocol of Qwen at the same time), so as to support the access of multiple heterogeneous models;

[0086] 4) Innovation of deep risk assessment capability

[0087] The keyword matching detection in the existing system cannot identify implicit risks (such as describing the preparation method of prohibited goods in the style of chemistry textbooks), and the present application realizes multi-dimensional analysis (joint judgment combining syntax features, semantic roles and sentiment polarity) and risk pattern mining (identifying "dangerous intent in legal expressions" by comparison and pre-training, such as associating "pesticide formula" with "human injury") based on enhanced models, so as to improve the deep risk assessment capability. BRIEF DESCRIPTION OF DRAWINGS

[0088] To more clearly illustrate the technical solutions in the embodiments of the present invention or the prior art, the accompanying drawings used in the description of the embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are merely some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without any creative effort.

[0089] Figure 1 This is a schematic diagram of the system of the present invention;

[0090] Figure 2 This is a schematic diagram of the method flow of the present invention.

[0091] Figure 3 For the present invention Figure 1 A detailed workflow diagram for tool management in China;

[0092] Figure 4 For the present invention Figure 1 Detailed workflow diagram of China's self-developed security system;

[0093] Figure 5 For the present invention Figure 1 A detailed flowchart of the self-iterative model process in the Chinese-developed security system. Detailed Implementation

[0094] To make the objectives, technical solutions, and advantages of the embodiments of the present invention clearer, the technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. All other embodiments obtained by those skilled in the art based on the embodiments of the present invention without creative effort are within the scope of protection of the present invention.

[0095] The following section describes the security and compliance assessment system based on a multimodal large model provided by this invention, using specific examples (such as...). Figure 1 As shown), the system components include:

[0096] The functional module cluster manages the various functional modules in the system and supports cross-modal consistency detection, which is used to compare the consistency of responses under different modalities.

[0097] Self-developed security system responsible for evaluating the execution of tasks, including adversarial test reinforcement, built-in multi-granularity data variation strategies (including synonym replacement, syntax reconstruction, intent confusion, etc.), improving evaluation robustness by simulating real attack scenarios, realizing end-to-end automation from data generation, model testing to risk assessment, supporting parameter configuration and result reproducibility, as well as self-iteration and version management of evaluation models, and the self-iteration mechanism is deeply combined with the compliance label system, which can continuously optimize the risk identification ability under the legal, ethical and industry norms.

[0098] I. Function module cluster is applicable to administrators, such as Figure 1 as shown, including:

[0099] User management, as the core of system permission control, is responsible for user role division (including evaluation group leader and evaluation personnel) and permission allocation. The evaluation group leader has the permission of tool uploading, data uploading, evaluation task creation and user management. The evaluation personnel have the permission of browsing tool library, executing evaluation task and viewing evaluation report. The permission verification mechanism based on role-based access control (RBAC) is used to ensure that different roles can only access authorized resources (such as evaluation group leader can configure tool verification rules, and evaluation personnel can only use verified tools to process data sets, etc.). At the same time, user operation logs (including evaluation task submission time, tool usage record, etc.) are recorded to provide support for audit and traceability;

[0100] Model management is used to solve the problem of multi-model compatibility. Through the plug-in protocol adaptation layer, the unified access and management of heterogeneous models are realized. The protocol conversion plug-in library is built in. In addition to supporting multiple interface types such as REST API, gRPC and SDK, it also supports protocol adaptation across modal input and output, ensuring the unified access and management of multi-modal models such as text, image and audio. The model library is maintained uniformly, and model version management is supported. Users can select the required model through the visual interface without worrying about the differences in underlying interfaces.

[0101] Tool management is the core of dynamic test set generation, as shown in Figure 3 provides a standardized interface to support users to upload custom data processing tools. Through the "tool uploading-verification-registration-calling" process, the tool chain is dynamically expanded, breaking through the limitations of traditional static data sets, and generating diversified test questions through dynamic tool chain generation.

[0102] Data management is responsible for the whole life cycle management of dynamic test sets. It supports users to upload data (supports batch import in formats such as Excel and Json, automatically parses data structure), custom extension and dynamic update. At the same time, combined with tool management, it can generate new dynamic test sets by processing the original data sets, and realize traceability through the association of original data set ID.

[0103] II. Self-developed security system is suitable for ordinary users, such as Figure 1 and Figure 4 , including:

[0104] Task execution, responsible for evaluating the whole process scheduling of the task, based on user configuration to automatically complete the "data processing - model interaction - result collection" process;

[0105] Evaluation report generation, integrating multi-source evaluation results to generate a comprehensive evaluation report;

[0106] Model self-iteration, realizing online iterative optimization of the evaluation model, and improving the risk identification ability through closed-loop learning.

[0107] 1) Task execution specifically includes:

[0108] Task creation: the user selects the model under test, the original data set, the processing tool and the evaluation model, and sets the evaluation parameters (such as the number of variants, the evaluation threshold, etc.);

[0109] Task distribution: allocate independent running environment for the evaluation task to avoid multi-task interference;

[0110] Data injection: call the processing tool to process the original data set to generate a dynamic test set;

[0111] Model interaction: through the protocol adaptation layer of model management, send the dynamic test set to the model under test, receive and store the answer results;

[0112] Progress monitoring: record the task progress in real time, and automatically retry when abnormal.

[0113] 2) Evaluation report generation specifically includes:

[0114] Evaluation index calculation: based on the evaluation results output by the evaluation model, calculate the detailed evaluation indexes including risk detection rate (such as correct identification rate of violation issues), misjudgment rate (such as the proportion of compliance answers marked as violation) and risk level distribution;

[0115] Multi-source comparison: compare the evaluation results of the evaluation model with the evaluation results output by the third-party model (such as OpenCompass, etc.), and mark the difference items;

[0116] Evaluation report output: support PDF / Word format, including detailed evaluation indexes, typical case analysis (such as high-risk answer original text and risk point marking, etc.) and improvement suggestions (such as risk types that the model needs to strengthen, etc.).

[0117] 3) Model self-iteration, as shown in Figure 5 , specifically includes:

[0118] Sample arrangement: Arrange the evaluated "question-answer-risk label" triplets into training set, validation set and test set;

[0119] Incremental training: Train and optimize the evaluation model using the training set and the validation set, and improve the training efficiency by dynamically adjusting the learning rate;

[0120] Effect verification: Use the test set to verify the performance of the new evaluation model. If the performance improvement is greater than the preset threshold, update the evaluation model library, otherwise roll back to the historical version;

[0121] Version management: Record the iteration history of the evaluation model (including training data volume, evaluation index change, etc.), and support version traceability.

[0122] On the basis of the above-mentioned disclosure of the safety compliance evaluation system based on the multi-modal large model, the specific process of the safety compliance evaluation method based on the multi-modal large model provided by the present application is introduced below in combination with specific examples (such as Figure 2 shown).

[0123] I. Task creation: The evaluation group leader creates an evaluation task, distributes it to the evaluators in real time, and monitors the specific evaluation process in real time to complete the closed-loop process of "task-execution-monitoring-result".

[0124] 1) Administrator login system: The administrator logs in to the system through the account password, and the system automatically verifies the identity information and permission level to ensure that the authorized administrator can enter the task management interface;

[0125] 2) Evaluation task creation (including specifying evaluators, models to be tested, requirements, etc.): The evaluation group leader fills in the basic information including task name, description and deadline in the task creation interface, selects evaluators from the user list and assigns specific responsibilities, and clearly defines the evaluation requirements;

[0126] 3) Evaluation task distribution: The system pushes the evaluation task details (including evaluation task ID, model information to be tested, time node, etc.) to the personal account of the designated evaluator in real time through the internal message pushing mechanism, and updates the task allocation status in the task board;

[0127] 4) Independent running environment allocation: The system creates an isolated running environment for each evaluator to avoid data interference and resource preemption when multiple users operate, and ensures the independence and stability of task execution;

[0128] 5) Real-time monitoring progress: The system collects the task execution data of each evaluator in real time through the process monitoring tool, and displays it in the form of visual charts (such as progress bar, percentage graph, etc.) on the evaluator console.

[0129] II. Data Processing: By constructing the full-process mechanism of "tool upload-verification-registration-invocation" and "data upload-variation-generation-storage", efficient generation and management of dynamic test sets are realized.

[0130] 1) Tool upload: Users upload custom tool programs through the standardized interface provided by the system;

[0131] 2) Verify tool availability: The system deploys the tool package to an isolated test environment and automatically executes preset test cases;

[0132] 3) Tool library registration: Validated tools are assigned a unique tool ID and associated with a function tag, stored in the tool library of the formal environment and made available to all users; tools that fail the verification will provide specific failure reasons (such as missing dependent libraries, incorrect output format, etc.) and support users to modify and re-upload;

[0133] 4) Upload original data set: Users upload structured original data sets, and the system supports batch import and automatically parses data fields, providing repair suggestions (such as filling default values, deleting invalid rows, etc.) for format errors (such as missing fields, etc.);

[0134] 5) Select tool chain (such as adversarial attack tools + semantic rewriting tools, etc.): Users select multiple tools in the tool library to form a tool chain, and adjust the execution order by dragging, the system provides tool compatibility prompts, and supports saving common tool chain templates for subsequent reuse;

[0135] 6) Variation processing: The tool chain executes the processing logic in order, and the adversarial attack tool generates induced variations (such as converting high-risk issues into hidden expressions, for example, changing "manufacturing bombs" to "household chemical pressure experiment plan") through synonym replacement, syntax reconstruction, and intent confusion, and records the processing results of each tool in real time;

[0136] 7) Generate dynamic test set: Aggregate all tool processing results, remove duplicate samples, and generate a dynamic test set containing original questions, variant questions, and corresponding tool IDs;

[0137] 8) Data storage: The dynamic test set is stored in the database, associated with the original data set ID, tool chain information, and generation time, and provides data export functions (supports CSV, JSON, etc.).

[0138] The data processing flow not only guarantees the safety and compatibility of tools, but also breaks through the limitations of traditional static data sets through diversified data variation, providing comprehensive and dynamically updated test data for large model safety and compliance evaluation.

[0139] III. Model Invoking: Through the construction of the whole process mechanism of "model selection-connection adaptation-prompt generation-interaction storage", the efficient invocation and result management of different types of tested models are realized.

[0140] 1) Selecting the tested model: the user selects the type of the tested model in the model list, and fills in the parameters such as interface address, access key and request frequency limit for API calling model; the local model needs to specify the model file path and running environment;

[0141] 2) Model connection (API calling model calls network service, local model loads base): the protocol adaptation layer automatically generates requests conforming to the target interface specification for API calling model, and configures the timeout retry mechanism; for local model, automatically load model base file, initialize inference environment, and detect whether the model supports batch inference to improve efficiency;

[0142] 3) Generating prompt words: the system generates standardized input text based on the variant problems in the dynamic test set and the preset prompt word template;

[0143] 4) Obtaining answer results: the asynchronous calling mode is used to send prompt words to the tested model, the request state is monitored in real time, task fragmentation processing is supported to avoid interface blocking; the answer results returned by the tested model (including text content, generation time consumption, token quantity, etc.) are received, and irrelevant formats (such as extra line breaks, <think>Tags, etc.);

[0144] 5) Store the answer result: store the answer result, and the information including the corresponding prompt words, the measured model ID, the request time and the processing status.

[0145] The process of calling the measured model not only breaks the interface barriers between different model architectures, ensuring the efficiency and stability of model calling, but also provides high-quality basic data support for subsequent risk assessment of the evaluation model through standardized processing and structured storage.

[0146] IV. Evaluation model calling: Through the construction of a "multi-model calling-depth analysis-risk assessment-model iteration" whole process mechanism, accurate evaluation of the measured model answer results and continuous optimization of the evaluation system are realized.

[0147] 1) Call multiple evaluation models: the system loads the selected model combination from the evaluation model library, and the self-developed evaluation model is deployed on the local server to ensure response speed, and the third-party model (such as GPT-4, DeepSeek, Qwen, etc.) is called through the API interface;

[0148] 2) Answer result analysis: multi-dimensional analysis (including semantic understanding, intent recognition, logical chain reasoning, etc.) of the answer results of the measured model;

[0149] 3) Comprehensive risk assessment level (including high, medium and low risk levels): generate an evaluation report according to the evaluation results output by the multiple evaluation models;

[0150] 4) Sample arrangement: the system automatically stores the information including the questions in the dynamic test set, the answer results of the measured model, and the risk labels corresponding to the evaluation results of the multiple evaluation models in the sample library according to time;

[0151] 5) Evaluation model self-training: samples are extracted from the sample library in proportion, the sample size is expanded through data enhancement technology, and the evaluation model is trained and optimized, and the training loss is monitored in real time to avoid overfitting;

[0152] 6) Model self-iteration: the new evaluation model needs to be verified for performance through the test set, and after verification, a new version number is generated, the old version model is replaced and the iteration log is recorded, and one-key rollback to the historical version is supported to deal with sudden situations.

[0153] The process of calling the evaluation model not only improves the accuracy of risk assessment through multi-model collaboration and multi-dimensional analysis, but also realizes the continuous evolution of evaluation ability through sample accumulation and model iteration mechanism, providing reliable and adaptive technical support for large model safety compliance evaluation.

[0154] V. Cross-modal consistency detection: In multi-modal including text, image and audio, the consistency of the answer content of the tested model is compared, and the compliance label system is combined for judgment.

[0155] The technical solutions of the present application, focusing on two core technologies of dynamic test set generation and multi-modal collaborative evaluation, construct a complete safety and compliance evaluation system and method based on multi-modal large models. At the system level, through the modular data processing tool chain, plug-in protocol adaptation layer and multi-modal risk assessment architecture, the technical effects of wide evaluation coverage, strong compatibility and deep analysis dimension are realized; at the workflow level, based on the dynamic test set generation of semantic conservation, the hierarchical collaborative evaluation mechanism and the online self-evolution logic, the problems of traditional methods relying on manual work, data rigidity and shallow evaluation are solved. System design and process innovation complement each other to form a quantifiable, scalable and adaptive large model safety and compliance evaluation closed-loop solution, providing an efficient and reliable standardized evaluation tool for the field of artificial intelligence safety.

[0156] The above embodiments are only used to illustrate the technical solutions of the present application, but not to limit them; although the present application has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that they can still modify the technical solutions recorded in the foregoing embodiments, or make equivalent replacement for part of the technical features; and these modifications or replacements will not make the essence of the corresponding technical solutions deviate from the spirit and scope of the technical solutions of the embodiments of the present application.< / think>

Claims

1. A system for security compliance assessment based on multi-modal large models, characterized in that: Comprise: A functional module cluster manages each functional module in the system and supports cross-modal consistency detection for comparing the consistency of answers under different modalities; A self-developed security system assesses the execution of tasks, including adversarial testing reinforcement, built-in multi-granularity data variation strategies, and simulation of real attack scenarios to improve evaluation robustness, achieving end-to-end automation from data generation, model testing to risk assessment, supporting parameter configuration and result reproducibility, as well as self-iteration and version management of evaluation models, and the self-iteration mechanism is deeply combined with the compliance label system, which can continuously optimize the risk identification ability under the legal, ethical and industry specifications; The self-developed security system is suitable for ordinary users, including: Task execution is responsible for the whole process scheduling of the evaluation task, and automatically completes the "data processing - model interaction - result collection" process based on user configuration; Evaluation report generation integrates multi-source evaluation results to generate a comprehensive evaluation report; Model self-iteration realizes online iteration optimization of evaluation models to improve risk identification ability through closed-loop learning; The task execution specifically includes: Task creation: the user selects the model under test, the original data set, the processing tool and the evaluation model, and sets the evaluation parameters; Task distribution: allocate independent running environment for evaluation task to avoid multi-task interference; Data injection: call the processing tool to process the original data set to generate a dynamic test set; Model interaction: through the protocol adaptation layer of model management, send the dynamic test set to the model under test, receive and store the answer results; Progress monitoring: record the task progress in real time, and automatically retry when abnormal; The data injection realizes efficient generation and management of dynamic test sets through the whole process mechanism of "tool upload - verification - registration - call" and "data upload - variation - generation - storage", including: S21, tool upload: the user uploads the custom tool program through the standardized interface provided by the system; S22, verify tool availability: the system deploys the tool package to an isolated test environment and automatically executes the preset test cases; S23, tool library registration: the verified tool will be assigned a unique tool ID and associated with a functional label, stored in the formal environment tool library and opened to all users; the tool that fails the verification will provide specific failure reasons, and the user can modify and re-upload it; S24, upload original data set: the user uploads the structured original data set, and the system supports batch import and automatically parses the data fields, providing repair suggestions for format errors; S25, select tool chain: the user selects multiple tools in the tool library to form a tool chain, and adjusts the tool execution order by dragging, the system provides tool compatibility prompts, and supports saving common tool chain templates for subsequent reuse; S26, variation processing: the tool chain executes the processing logic in order, and the adversarial attack tool generates induced variations through synonym replacement, syntax reconstruction and intent confusion, and records the processing results of each tool in real time; S27, generating a dynamic test set: aggregating the processing results of all tools, removing duplicate samples, and generating a dynamic test set containing original questions, variant questions, and corresponding tool IDs; S28, data storage: the dynamic test set will be stored in the database, associated with the original dataset ID, tool chain information, and generation time, while providing data export functions.

2. The multimodal large model-based security compliance assessment system according to claim 1, wherein: The functional module cluster is suitable for administrators and includes user management, model management, tool management, and data management; In addition to supporting multiple interface types such as REST API, gRPC, and SDK, the model management also supports protocol adaptation for cross-modal input and output, ensuring unified access and management of multi-modal models such as text, images, and audio.

3. The multimodal large model based security compliance assessment system of claim 1, wherein: The evaluation report generation specifically includes: Evaluation index calculation: based on the evaluation results output by the evaluation model, calculate detailed evaluation indexes such as risk detection rate, false positive rate, and risk level distribution; Multi-source comparison: compare the evaluation results of the evaluation model with the evaluation results output by third-party models, and mark the differences; Evaluation report output: support PDF / Word format, including detailed evaluation indexes, typical case analysis, and improvement suggestions; The model self-iteration specifically includes: Sample arrangement: arrange the evaluated "question-answer-risk label" triplets into training set, validation set, and test set; Incremental training: train and optimize the evaluation model using the training set and validation set, and improve training efficiency by dynamically adjusting the learning rate; Effect verification: verify the performance of the new evaluation model using the test set. If the performance improvement is greater than the preset threshold, update the evaluation model library, otherwise roll back to the historical version; Version management: record the iteration history of the evaluation model, support version traceability.

4. The method for security compliance assessment based on a multi-modal large model, applied to the system for security compliance assessment based on a multi-modal large model according to claim 1, characterized in that: The following steps are included: S1, task creation; S2, data processing; S3, calling the model under test; S4, calling the evaluation model; S5, cross-modal consistency detection: in multi-modal such as text, image and audio, the consistency of the answer content of the model under test is compared, and the compliance label system is used for judgment.

5. The method of claim 4, wherein: S1, task creation: the evaluation group leader creates an evaluation task, distributes it to the evaluation personnel in real time, and monitors the specific evaluation process in real time, completing the closed-loop process of "task-execution-monitoring-result", including: S11, administrator login system: the administrator logs in to the system through the account password, and the system automatically verifies the identity information and permission level to ensure that authorized administrators can enter the task management interface; S12, evaluation task creation: the evaluation group leader fills in the basic information such as task name, description, and deadline in the task creation interface, selects the evaluation personnel from the user list and assigns specific responsibilities, and clearly defines the evaluation requirements; S13, evaluation task distribution: the system pushes the evaluation task details to the designated evaluation personnel's personal account in real time through the internal message pushing mechanism, and updates the task allocation status in the task board; S14, independent running environment allocation: the system creates an isolated running environment for each evaluation personnel to avoid data interference and resource occupation when multiple users operate, ensuring the independence and stability of task execution; S15, Real-time monitoring progress: The system collects the task execution data of each evaluator in real time through the process monitoring tool and displays it in the form of a visual chart on the evaluation team leader console.

6. The method of claim 5, wherein the method further comprises: S2 Data processing: Through the construction of a "tool upload-verification-registration-call" and "data upload-variant-generation-storage" full-process mechanism, efficient generation and management of dynamic test sets are achieved, including: S21, Tool upload: Users upload custom tool programs through the standardized interface provided by the system; S22, Verify tool availability: The system deploys the tool package to an isolated test environment and automatically executes pre-set test cases; S23, Tool library registration: Tools that pass the verification will be assigned a unique tool ID and associated with a function tag, stored in the tool library in the formal environment and made available to all users; Tools that fail the verification will provide specific failure reasons, allowing users to modify and re-upload; S24, Upload original data set: Users upload structured original data sets, and the system supports batch import and automatically parses data fields, providing repair suggestions for format errors; S25, Select tool chain: Users select multiple tools in the tool library to form a tool chain and adjust the tool execution order through drag and drop. The system provides tool compatibility prompts and supports saving common tool chain templates for future reuse; S26, Variant processing: The tool chain executes processing logic in order, and the anti-attack tool generates induced variants through synonym replacement, syntax reconstruction, and intent obfuscation, and records the processing results of each tool in real time; S27, Generate dynamic test set: Aggregate all tool processing results, remove duplicate samples, and generate a dynamic test set containing original questions, variant questions, and corresponding tool IDs; S28, Data storage: The dynamic test set is stored in the database, associated with the original data set ID, tool chain information, and generation time, and provides data export functions.

7. The method of claim 6, wherein: S3 Call the measured model: Through the construction of a "model selection-connection adaptation-prompt generation-interaction storage" full-process mechanism, efficient calling and result management of different types of measured models are achieved, including: S31, Select the measured model: Users select the type of measured model in the model list. For API call models, fill in parameters such as interface address, access key, and request frequency limit; for local models, specify the model file path and running environment; S32, Model connection: The protocol adaptation layer automatically generates requests that meet the target interface specifications for API call models and configures a timeout retry mechanism; for local models, automatically load model base files, initialize the inference environment, and detect whether the model supports batch inference to improve efficiency; S33, Generate prompt words: The system generates standardized input text based on variant questions in the dynamic test set and pre-set prompt word templates; S34, Get answer results: Use asynchronous calling to send prompt words to the measured model, monitor request status in real time, support task sharding to avoid interface blocking; receive the answer results returned by the measured model and automatically remove irrelevant formats; S35, store the answer result: store the answer result and the information including the corresponding prompt word, the measured model ID, the request time and the processing status.

8. The method of claim 7, wherein: S4, evaluation model calling: through the construction of the whole process mechanism of "multi-model calling-depth analysis-risk assessment-model iteration", the accurate evaluation of the measured model answer result and the continuous optimization of the evaluation system are realized, including: S41, calling multi-evaluation model: the system loads the selected model combination from the evaluation model library, the self-developed evaluation model is deployed in the local server to ensure the response speed, and the third-party model is called through the API interface; S42, answer result analysis: multi-dimensional analysis is performed on the answer result of the measured model; S43, comprehensive evaluation of risk level: according to the evaluation results output by the multi-evaluation model, the evaluation report is integrated and generated; S44, sample arrangement: the system automatically stores the information including the questions in the dynamic test set, the answer result of the measured model, and the risk label corresponding to the evaluation result of the multi-evaluation model in the sample library according to time; S45, evaluation model self-training: samples are extracted from the sample library in proportion, the sample size is expanded through data enhancement technology, and the evaluation model is trained and optimized, and the training loss is monitored in real time to avoid overfitting; S46, model self-iteration: the new evaluation model needs to be verified for performance through the test set, and after the verification, a new version number is generated, the old version model is replaced and the iteration log is recorded, and one-key rollback to the historical version is supported to deal with unexpected situations.

Images