Large model application security management and control platform

Abstract

The application discloses a large model application security management and control platform, and relates to the field of large model security application management and control.The platform comprises an MCP protection module, an RAG protection module, an API interface protection module, a secure link module, a gateway application protection module, a data set protection module, a security test evaluation module, a security audit module and a security monitoring and early warning module.The MCP protection module is used for providing security protection when MCP external tools are dispatched.The RAG protection module is used for providing security protection when RAG external data is retrieved.The API interface protection module is used for providing security protection when API interfaces are called.The secure link module is used for providing security protection when data is transmitted.The gateway application protection module is used for providing security protection when a dialogue question and answer is performed.The data set protection module is used for providing data security detection when fine tuning is performed.The security test evaluation module is used for evaluating the large model.The security audit module is used for performing whole-process log auditing on each module.The security monitoring and early warning module is used for performing real-time monitoring on each module, and an early warning is sent when an anomaly is found.The application can provide security protection functions in the large model application process.

Description

Technical Field

[0001] This application relates to the field of large-scale model security application management technology, and in particular to a large-scale model application security management platform. Background Technology

[0002] In recent years, large-scale modeling technologies, represented by generative artificial intelligence, have demonstrated powerful capabilities in fields such as natural language processing, image generation, and intelligent dialogue. They have been widely applied in scenarios such as intelligent customer service, content creation, and data analysis, driving the digital transformation and intelligent upgrading of various industries. However, with the rapid expansion of large-scale modeling applications, their security risks have gradually been exposed, attracting widespread attention from all sectors of society.

[0003] Large-scale models face multiple security threats at both the input and output stages. During the input phase, malicious users may use carefully crafted prompts to induce the model to generate illegal content, including but not limited to false information, illegal statements, and sensitive information. They may even use prompt injection attacks to bypass model security restrictions, obtain illegal data, or control model behavior. During the output phase, the content generated by the large-scale model may contain discriminatory, misleading, or infringing information due to training data biases, algorithmic defects, etc., leading to legal risks and reputational damage for users. Furthermore, large-scale models involve the processing of large amounts of user data; if sensitive data is leaked or misused, it will seriously infringe upon personal privacy and corporate trade secrets.

[0004] Furthermore, access control for large-scale models also presents security vulnerabilities. A lack of effective access control can lead to unauthorized access, data misuse, and other issues, threatening data security and system stability. Existing security measures are fragmented and fail to cover the entire process from request access to content output, thus failing to form a complete security protection loop and struggling to cope with complex and ever-changing security threats. Summary of the Invention

[0005] The purpose of this application is to provide a security management platform for large-scale model applications, which can provide security protection functions during the application of large-scale models.

[0006] To achieve the above objectives, this application provides the following solution.

[0007] This application provides a large-scale application security management platform, which includes: an MCP protection module, a RAG protection module, an API interface protection module, a security link module, a gateway application protection module, a dataset protection module, a security testing and evaluation module, a security audit module, and a security monitoring and early warning module.

[0008] The MCP protection module is used to provide security protection for large models when scheduling external MCP tools.

[0009] The RAG protection module is configured to provide security protection when the large model performs RAG external data retrieval.

[0010] The API interface protection module is configured to provide security protection when the large model performs API interface calling.

[0011] The secure link module is configured to provide security protection when the large model performs data transmission.

[0012] The gateway application protection module is configured to provide security protection when the large model performs dialogue question answering.

[0013] The dataset protection module is configured to provide data security detection when the large model performs fine-tuning.

[0014] The security test evaluation module is configured to evaluate the function, performance and security of the large model after training or fine-tuning.

[0015] The security audit module is configured to perform full-process log auditing on data interaction and operation behavior in each module.

[0016] The security monitoring and early warning module is configured to monitor the MCP protection module, the RAG protection module, the API interface protection module, the secure link module, the gateway application protection module, the dataset protection module and the security audit module in real time, and issue a warning when an anomaly is found.

[0017] Optionally, the MCP protection module comprises the following sub-modules.

[0018] The service unified authentication sub-module is configured to traverse public MCP service resources based on a crawler technology, parse MCP service configuration files through a Python technology, add a unique representation, a detailed description and a function attribute of the MCP service configuration files according to a service function, and establish an MCP service list.

[0019] The MCP sandbox sub-module is configured to construct a malicious behavior rule library and analyze malicious MCP services according to the malicious behavior rule library.

[0020] The MCP calling control sub-module is configured to collect calling conditions of each MCP service in real time based on a unique identifier of the MCP service, and record key information of each calling; the key information includes time, an initiator, calling parameters and a return result.

[0021] The operation permission control sub-module is configured to establish an API Key of a user based on a user identity, construct a resource access rule in combination with a user attribute, a resource attribute and an environment condition, encrypt a permission parameter through an encryption algorithm, judge a user permission through verification of the permission parameter, and form a dynamic permission allocation mechanism.

[0022] A MCP protocol security guarantee submodule is configured to guarantee the security of the MCP protocol.

[0023] Optionally, the RAG protection module comprises the following submodules.

[0024] A data source security management submodule is configured to build systematic data source security management capabilities from two aspects of trust mechanisms and sensitive content identification.

[0025] A classification and grading management submodule is configured to assign labels to documents and form metadata descriptions based on a document classification system based on keyword matching, regular expressions and semantic similarity analysis, and in combination with multi-dimensional sensitive levels.

[0026] A permission security optimization submodule is configured to adopt a hybrid access control strategy combining role-based access control and attribute-based access control to enhance the flexibility and context awareness of access control while ensuring the granularity of permissions.

[0027] Optionally, the API interface protection module comprises the following submodules.

[0028] A multi-level authentication submodule is configured to implement multi-level authentication.

[0029] A traffic monitoring submodule is configured to record user information, request parameters, response results, time consumption and error codes of each API call, and to perform alarm processing on abnormal behaviors detected according to preset alarm rules.

[0030] Optionally, the secure link module comprises the following submodules.

[0031] A Token secure link submodule is configured to encrypt data links covering Token transmission through a secure transmission protocol.

[0032] A call situation monitoring submodule is configured to monitor the calling situation of Tokens in real time.

[0033] Optionally, the data set protection module comprises the following submodules.

[0034] A data set risk identification submodule is configured to identify data poisoning, sensitive data leakage and backdoor attacks in data sets.

[0035] A data processing submodule is configured to clean and verify the identified data.

[0036] Optionally, the security test evaluation module comprises the following submodules.

[0037] A forbidden word library submodule is configured to build a forbidden word library.

[0038] The dictionary update submodule is used to update the prohibited dictionary.

[0039] The deep-induced attack submodule is used to conduct "forbidden word" jailbreak tests on large models using a multi-step progressive induction strategy; the multi-step progressive induction strategy includes: seed question bank generation, multi-step induction loop, and termination output.

[0040] The model security assessment submodule is used to systematically evaluate the security protection capabilities of large models in different application scenarios through static analysis, dynamic attack verification, multiple attack simulation strategies, and output of multi-dimensional security assessment coefficients. The multiple attack simulation strategies include: prohibiting sample injection, backdoor triggering test, data poisoning verification, and context-induced bypass. The multi-dimensional security assessment coefficients include: response accuracy rate, violation response rate, security shielding trigger rate, jailbreak success rate, and security risk index.

[0041] The gap analysis submodule is used to benchmark against current national laws and regulations and the "Basic Requirements for Security of Generative Artificial Intelligence Services", analyze the violations and non-compliance of the large model test answers, and adjust the security protection strategy and conduct model fine-tuning training by combining the results of deep provocation attacks and security assessments.

[0042] Optionally, the gateway application protection module includes the following sub-modules.

[0043] The scanning submodule is used to perform real-time scanning of input and output content.

[0044] The filtering submodule is used to filter input and output content.

[0045] Optionally, the security monitoring and early warning module includes the following sub-modules.

[0046] The risk monitoring submodule is used to perform real-time monitoring of the MCP protection module, the RAG protection module, the API interface protection module, the security link module, the gateway application protection module, the dataset protection module, and the security audit module.

[0047] The early warning submodule is used to issue early warnings when an anomaly is detected.

[0048] Optionally, the security audit module includes the following sub-modules.

[0049] The security log aggregation and analysis submodule is used to aggregate security logs from various modules and analyze those security logs.

[0050] The abnormal event alarm submodule is used to promptly issue abnormal event alarms based on the analysis results of security logs.

[0051] Based on the specific embodiments provided in this application, the following technical effects are disclosed.

[0052] This application provides a large-scale model application security management platform, which includes: an MCP protection module, a RAG protection module, an API interface protection module, a secure link module, a gateway application protection module, a dataset protection module, a security testing and evaluation module, a security audit module, and a security monitoring and early warning module. Through the cooperation of these modules, security protection can be provided for large-scale model applications, covering the entire lifecycle of large-scale models. They can work independently to provide targeted security protection, and also collaborate with each other. This large-scale model application security management platform has targeted protection modules for multiple scenarios, with strong scenario adaptability. It integrates data detection, model evaluation, and rectification guidance functions during the fine-tuning stage, forming a closed-loop management system. Security audits are tracked throughout the process, monitoring and early warning are monitored in real time, and anomaly responses are timely. The modules collaboratively construct a complete protection system, effectively preventing and controlling various security risks. Attached Figure Description

[0053] To more clearly illustrate the technical solutions in the embodiments of this application or the prior art, the drawings used in the embodiments will be briefly introduced below. Obviously, the drawings described below are only some embodiments of this application. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0054] Figure 1 This is an overall framework diagram of a large-scale application security management platform provided in one embodiment of this application.

[0055] Figure 2 This is a functional relationship logic diagram of a large-scale application security management platform provided in one embodiment of this application. Detailed Implementation

[0056] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of this application, and not all embodiments. Based on the embodiments of this application, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of this application.

[0057] The problems that exist in this field are as follows.

[0058] (1) Lack of security management tools or platforms for large models.

[0059] (2) Some tools can provide security detection and protection for the data security (output content) of large models, but lack protection for large model security testing and evaluation, MCP, API, dataset, RAG, etc.

[0060] (3) Existing security products or platforms have relatively simple functions, mainly focusing on security protection in a certain aspect, and lack consideration for security protection throughout the entire process of large-scale model operation and application.

[0061] (4) The security products or platforms available lack comprehensive consideration for multi-faceted security protection and mutual cooperation, resulting in poor applicability or ineffectiveness.

[0062] To address the aforementioned issues, this application establishes a large-scale model security application management and control platform to ensure the secure application of large-scale models.

[0063] To make the above-mentioned objectives, features and advantages of this application more apparent and understandable, the application will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0064] In one exemplary embodiment, such as Figure 1 and Figure 2 As shown, a large-scale application security management platform is provided, which includes: an MCP protection module, a RAG protection module, an API interface protection module, a security link module, a gateway application protection module, a dataset protection module, a security testing and evaluation module, a security audit module, and a security monitoring and early warning module.

[0065] The MCP protection module is used to provide security protection for large models when scheduling external MCP tools.

[0066] The RAG protection module is used to provide security protection for large models when performing RAG external data retrieval.

[0067] The API interface protection module is used to provide security protection for large models when making API interface calls.

[0068] The secure link module is used to provide security protection for large models during data transmission.

[0069] The gateway application protection module is used to provide security protection for large models when conducting dialogue and question answering.

[0070] The dataset protection module is used to provide data security detection when large models are being fine-tuned.

[0071] The security testing and evaluation module is used to evaluate the functionality, performance, and security of the large model after training or fine-tuning.

[0072] The security audit module is used to perform full-process log auditing of data interactions and operational behaviors in each module.

[0073] The security monitoring and early warning module is used to monitor the MCP protection module, the RAG protection module, the API interface protection module, the security link module, the gateway application protection module, the dataset protection module, and the security audit module in real time, and issue an early warning when an anomaly is detected.

[0074] Specifically, the MCP protection module, RAG protection module, API interface protection module, secure link module, and gateway application protection module provide security protection for large models in different application scenarios, such as API interface calls, MCP external tool scheduling, RAG external data retrieval, data transmission, and dialogue question answering. The dataset protection module provides data security detection during the large model fine-tuning phase. The security testing and evaluation module is responsible for evaluating the functionality, performance, and security of the trained or fine-tuned large model, outputting security evaluation coefficients in real time, and performing gap analysis to guide further rectification and hardening of the large model. The security audit module runs through all large model application scenarios, performing full-process log auditing of data interactions and operational behaviors in each module. The security monitoring and early warning module monitors all the above protection and audit modules in real time, promptly detecting anomalies and issuing early warnings, thus forming a comprehensive security protection system. Through the cooperation between the modules, various security risks during model application are prevented and controlled. A detailed introduction to each module follows.

[0075] 1 MCP protection module.

[0076] The MCP protection module constructs an end-to-end security management system for the entire lifecycle of MCP services through five core modules: unified authentication, sandbox detection, call control, access management, and protocol security assurance. The system adopts a microservice architecture, supports horizontal scaling and component-based deployment, and can effectively identify, prevent, and defend against various security threats targeting MCP services.

[0077] As an optional implementation, the MCP protection module includes the following sub-modules.

[0078] 1.1 Service Unified Authentication Submodule.

[0079] The unified service authentication submodule is used to traverse publicly available MCP service resources such as code hosting platforms, MCP aggregation platforms, and technical communities based on crawler technology. It parses the MCP service configuration files using Python technology, adds unique representations, detailed descriptions, and functional attributes to the MCP service configuration files according to the service functions, and establishes an MCP service list.

[0080] 1.2 MCP Sandbox Module.

[0081] The MCP sandbox module is used to build a malicious behavior rule base and analyze malicious MCP services based on the malicious behavior rule base.

[0082] Using virtualization technology and Docker container technology, we built an MCP security service sandbox that runs on different operating systems such as Linux and Windows. We constructed a malicious behavior rule base, checked the legality of the digital signature and compilation information of the MCP service through file feature matching, captured the program's low-level calls to the operating system through Hook technology, and monitored network and resource usage. Based on the rule base, we analyzed malicious MCP services.

[0083] 1.3 MCP call control submodule.

[0084] The MCP call control submodule is used to collect call information for each MCP service in real time based on the service's unique identifier. It records key information such as the call time, initiating entity, call parameters, and return result for each call. It uses regular expressions and type checks to verify the validity of input parameters and promptly detects and blocks erroneous calls caused by similar functions or names.

[0085] 1.4 Operation permission control submodule.

[0086] The operation permission control submodule is used to establish a unique API key for each user based on their identity, construct resource access rules by combining user attributes, resource attributes, and environmental conditions, and encrypt permission parameters using encryption algorithms such as RSA. By verifying the permission parameters, user permissions are determined, forming a dynamic permission allocation mechanism.

[0087] 1.5 MCP Protocol Security Assurance Submodule.

[0088] The MCP protocol security protection submodule is used to ensure the security of the MCP protocol.

[0089] 1) Establish a two-way TLS authentication channel and use high-strength encryption suites such as ECDHE, ECDSA, and AES256 to encrypt and protect API keys and user data; adopt an instruction whitelist mechanism to allow only predefined secure instruction sets and perform boundary checks on instruction parameters to prevent buffer overflows; protect against instruction replay attacks by using serial number increment and timestamp verification, and use the CRC32 check algorithm to ensure data integrity.

[0090] 2) Use temporary tokens or anonymous identifiers to avoid carrying sensitive characters.

[0091] 3) The EncryptMessage (regular) function is used to encrypt session messages and isolate the context session. Cookie and session clearing technologies ensure that the context data of different users / sessions are not visible to each other.

[0092] 2. RAG protection module.

[0093] The RAG protection module plays a crucial role in monitoring and controlling the risks associated with external knowledge bases or knowledge augmentations of large models. This module accurately identifies and addresses risks that RAG may cause, such as data privacy leaks, factual errors, malicious search attacks, content bias, and system abuse.

[0094] As an optional implementation, the RAG protection module includes the following sub-modules.

[0095] 2.1 Data Source Security Management Submodule.

[0096] The data source security management submodule is used to build a systematic data source security management capability from two levels: trust mechanism and sensitive content identification.

[0097] To ensure the security of data sources when RAG technology accesses external knowledge bases, this module has built a systematic data source security control capability from two levels: trust mechanism and sensitive content identification.

[0098] First, before data is integrated, a data source trust list is established, allowing only data sources that have passed manual review or automated security auditing to be added to the whitelist, preventing unauthorized or potentially malicious data access from the source. Simultaneously, all accepted knowledge base documents must generate digital signatures and version numbers during the integration phase, employing the SHA-256 hash algorithm and public key signature mechanism to ensure content integrity and source trustworthiness. When a document is accessed or updated, the legality of the signature and version consistency are automatically verified. If signature errors, version conflicts, or forgery are detected, the integration operation will be suspended and the relevant risk event will be recorded to prevent malicious tampering of documents for use in large models.

[0099] Secondly, regarding content security, a multi-dimensional sensitive keyword library is pre-built and maintained, covering high-risk words such as sensitive user personal identity information, key technical details and parameters, malicious commands, and expressions of ideological bias. The module employs regular expression matching and Aho-Corasick multi-pattern matching algorithms to automatically batch scan structured and unstructured documents in the knowledge base, accurately identifying potential sensitive content. For text fragments that match the rules, they will be automatically marked, blocked, or alerted according to preset strategies, effectively preventing harmful content from entering the RAG retrieval path and ensuring information compliance, security, and trustworthiness during the use of the large model.

[0100] The deployment of this module effectively builds the first line of defense in the RAG data retrieval process, minimizing the risks of source data pollution, sensitive information leakage, and model misleading.

[0101] 2.2 Classification and Grading Control Submodule.

[0102] The classification and hierarchical control submodule is used for a document classification system based on keyword matching, regular expressions, and semantic similarity analysis. It combines multi-dimensional sensitivity levels to assign tags to documents and form metadata descriptions.

[0103] A document classification system based on keyword matching, regular expressions, and semantic similarity analysis, combined with multi-dimensional sensitivity levels (personal identity information, financial data, strategic information, legal compliance, and internal processes, etc.), assigns tags to documents and forms metadata descriptions. Fine-grained access control is achieved through user permission groups and document tags, effectively protecting sensitive information assets.

[0104] 2.3 Permission and security optimization submodule.

[0105] The permission security optimization submodule is used to adopt a hybrid access control strategy that combines role-based access control and attribute-based access control, which enhances the flexibility and context awareness of access control while ensuring the granularity of permissions.

[0106] This module adopts a hybrid access control strategy that combines role-based access control with attribute-based access control, which enhances the flexibility and context awareness of access control while ensuring the granularity of permissions.

[0107] In role-based access control, users are assigned to different roles based on their identity and responsibilities, such as data labelers, model developers, and auditors, and fixed permission boundaries are bound to each role. In attribute-based access control, dynamic factors such as user attributes (e.g., department, job level), resource attributes (e.g., document sensitivity level, knowledge type), and access environment attributes (e.g., request time, device type, IP address) are further introduced to construct multi-dimensional access judgment logic, enabling dynamic authorization of access behavior under specific business scenarios.

[0108] To ensure the secure transmission and reliable verification of permission parameters, encryption and signature mechanisms are introduced into the access control chain. Specifically, asymmetric encryption algorithms such as RSA are used to encrypt and protect sensitive fields such as permission tokens and access parameters, and digital signature technology is combined to verify the integrity of transmitted content and prevent permission forgery or tampering. The system uses a public key to decrypt and verify the signature in the access request, ensuring the authenticity and trustworthiness of the caller's identity and permission parameters.

[0109] This module provides a fine-grained and dynamic access control mechanism for the access process of external knowledge base resources in RAG applications, preventing the leakage of sensitive information caused by unauthorized or unauthorized access.

[0110] 3. API Interface Protection Module.

[0111] This module accurately identifies and handles risks that API interfaces may cause, such as data leakage, unauthorized calls, injection attacks, and service abuse.

[0112] As an optional implementation, the API interface protection module includes the following sub-modules.

[0113] 3.1 Multi-level authentication submodule.

[0114] The multi-level authentication submodule is used to implement multi-level authentication.

[0115] This module builds a four-layer authentication system based on the application layer. The first layer, basic authentication, uses API Key authentication, assigning a unique API Key and Secret to each client and using HMAC signatures to verify the integrity of request parameters. The second layer, identity verification, uses OAuth 2.0 / OpenID Connect to support third-party login, issuing short-term access tokens and refresh tokens, and using JWT tokens to store user identity information and permission claims. The third layer, access control, defines user roles and corresponding API access permissions, dynamically authorizing based on user attributes, resource attributes, and environmental conditions. The fourth layer, data-level protection, controls the API response fields that users can access and filters the scope of data access based on user identity.

[0116] 3.2 Traffic monitoring submodule.

[0117] The traffic monitoring submodule is used to record user information, request parameters, response results, time consumption and error codes for each API call, and to handle alarms for abnormal behaviors detected according to preset alarm rules.

[0118] 4. Security Link Module.

[0119] The secure link module undertakes the critical task of protecting the data transmission link security in large-scale model applications, providing security assurance for the entire large-scale model application process.

[0120] As an optional implementation, the secure link module includes the following sub-modules.

[0121] 4.1 Token Security Link Submodule.

[0122] The Token Secure Link submodule is used to encrypt the data link covering the transmission of tokens through a secure transmission protocol.

[0123] This module encrypts the data link covering token transmission through secure transmission protocols such as HTTPS, ensuring the confidentiality and integrity of token transmission and managing the token status of large models to safeguard enterprise data security.

[0124] 4.2 Call status monitoring submodule.

[0125] The call monitoring submodule is used to monitor the call status of Tokens in real time.

[0126] 5. Dataset protection module.

[0127] The dataset protection module undertakes the critical tasks of risk identification and data processing for the dataset. Based on existing data processing tools (Adversarial Robustness Toolbox (ART) for data poisoning detection; Microsoft Presidio for sensitive data detection, masking, and anonymization; and a local density-based detector for backdoor attack sample detection), it detects data security issues such as data poisoning, sensitive data leakage, and backdoor attacks in the dataset. It also performs cleaning and verification operations on malicious data to ensure the security and trustworthiness of third-party data, laying a solid foundation for subsequent model fine-tuning and training.

[0128] As an optional implementation, the dataset protection module includes the following sub-modules.

[0129] 5.1 Dataset Risk Identification Submodule.

[0130] The dataset risk identification submodule is used to identify data poisoning, sensitive data leakage, and backdoor attacks in the dataset.

[0131] 5.2 Data Processing Submodule.

[0132] The data processing submodule is used to clean and verify the identified data.

[0133] 6. Security testing and evaluation module.

[0134] The security testing and evaluation module is responsible for evaluating the functionality, performance, and security of the large model after training or fine-tuning. It outputs security evaluation coefficients in real time, performs gap analysis, and guides the large model to further improve and strengthen its functionality, performance, and security.

[0135] As an optional implementation, the security testing and evaluation module includes the following sub-modules.

[0136] 6.1 Disable the dictionary submodule.

[0137] The Forbidden Words submodule is used to build a forbidden word library.

[0138] A basic vocabulary database was created using structured and unstructured databases. Illegal and non-compliant content as stipulated by current Chinese laws and regulations was collected and organized, as well as sensitive words on the Internet. A prohibited vocabulary was constructed, and structured prohibited words were placed into the structured database and unstructured prohibited words were placed into the unstructured database to form a basic prohibited vocabulary database.

[0139] 6.2 Dictionary Update Submodule.

[0140] The dictionary update submodule is used to update the prohibited dictionary.

[0141] 1) Collect sensitive words on the Internet by automatically crawling Internet data, compare them with the content in the original word database and update them incrementally.

[0142] 2) Disable updates to the dictionary by manually adding entries.

[0143] 6.3 Deep Induced Attack Submodule.

[0144] The deep-induced attack submodule is used to conduct "forbidden word" jailbreak tests on large models using a multi-step progressive induction strategy; the multi-step progressive induction strategy includes: seed question bank generation, multi-step induction loop, and termination output.

[0145] This module employs a multi-step progressive induction strategy to conduct "forbidden word" jailbreak tests on the model, systematically and automatically identifying the model's security boundaries and vulnerabilities under complex prompt chains. The overall process consists of three main stages: seed question bank generation, multi-step induction loop, and termination output.

[0146] The first stage involves generating a seed question bank. The system pre-defines a high-risk dictionary and, combined with prompt injection attack templates, automatically synthesizes the first batch of provocation questions using a large language model (LLM) and template transformation techniques. These questions are assigned initial risk labels, forming the 0th round test set, laying the foundation for subsequent progressive testing.

[0147] The second stage is a multi-step induced cycle. Each cycle contains four key elements.

[0148] (1) Perform test - Input the current question into the model to be tested, record the response and the triggering of the security policy, and use OpenAIModeration, PerspectiveScore or self-developed classifier to classify the risk of the answer.

[0149] (2) Decision branching - If the model outputs the highest risk answer or triggers the safety shield, the generation of this branch will be terminated; otherwise, the next round will begin.

[0150] (3) Question evolution - using methods such as LLM semantic rewriting, keyword exchange, genetic algorithm, and MCTS search, answers that do not trigger high risk are "mutated" and "recombined" to generate more persuasive questions for the next round.

[0151] (4) Risk accumulation - In each round of questions, add context or "ignore rules" and other instruction chains to simulate real hacker multi-step social engineering attacks and gradually increase the jailbreak intensity.

[0152] When a certain inducement chain reaches the set maximum depth or the generated high-risk questions reach the threshold, the system enters the third stage—termination and output. The module outputs a list of successful jailbreak samples (inducement questions and dangerous model answers), a jailbreak difficulty curve (rounds vs. risk scores), security shield trigger rate, and the shortest jailbreak path, among other quantitative indicators, to intuitively present the model's defense weaknesses.

[0153] Through an iterative generation + risk feedback testing mechanism, the deep learning attack module can quickly build a progressively enhanced high-risk warning word library, enabling comprehensive and in-depth testing of large-scale jailbreak risks and significantly improving the ability to detect complex attack paths.

[0154] 6.4 Model safety assessment submodule.

[0155] The model security assessment submodule is used to systematically evaluate the security protection capabilities of large models in different application scenarios through static analysis, dynamic attack verification, multiple attack simulation strategies, and output of multi-dimensional security assessment coefficients. The multiple attack simulation strategies include: prohibiting sample injection, backdoor triggering test, data poisoning verification, and context-induced bypass. The multi-dimensional security assessment coefficients include: response accuracy rate, violation response rate, security shielding trigger rate, jailbreak success rate, and security risk index.

[0156] This module conducts a systematic evaluation of the security protection capabilities of large models in different application scenarios, comprehensively covering their functional reliability, performance stability, and security protection capabilities across multiple dimensions, including MCP applications, RAG access, and API calls.

[0157] First, automated vulnerability scanning tools (such as OpenVAS, Nessus, ZAP, etc.) are used to perform static analysis on the large model deployment environment and related service interfaces to identify common configuration vulnerabilities, dependency vulnerabilities, authentication bypass and other system weaknesses. Then, combined with penetration testing techniques, attacker behavior is simulated to perform dynamic attack verification on the system, including session hijacking at the network security layer, unauthorized access at the access control layer, and prompt injection unique to large models, to comprehensively evaluate the protection capabilities of the model and interfaces under multi-dimensional attack scenarios.

[0158] At the dynamic interaction level, multiple attack simulation strategies (including sample injection prohibition, backdoor trigger testing, data poisoning verification, context-induced bypass, etc.) and the deep-induced attack submodule in 6.3 are integrated to conduct security tests on the large model after training or fine-tuning, and evaluate its security defense capabilities, sensitive information exposure risks and content compliance from different attack paths.

[0159] Furthermore, the module outputs security assessment coefficients across multiple dimensions in real time, such as response accuracy, violation response rate, security blocking trigger rate, jailbreak success rate, and security risk index. Based on these multi-dimensional security assessment indicators, and through normalization and custom weighting strategies, a unified security assessment coefficient is comprehensively constructed. This coefficient serves as a centralized representation of the security capabilities of a large model. It not only supports quantitative comparison and trend analysis of the security performance of different model versions (such as before and after fine-tuning, and between training epochs), but also serves as an important basis for evaluating the effectiveness of security hardening and verifying the effect of strategy optimization. It helps identify the evolution path of the model's security capabilities during training / deployment, continuously improving its robustness and reliability.

[0160] 6.5 Gap Analysis Submodule.

[0161] The gap analysis submodule is used to benchmark against current national laws and regulations and the "Basic Requirements for Security of Generative Artificial Intelligence Services", analyze the violations and non-compliance of the large model test answers, and adjust the security protection strategy and conduct model fine-tuning training by combining the results of deep provocation attacks and security assessments.

[0162] 1) By benchmarking against current national laws and regulations and using the graph indexing method, we can further analyze the non-rejected but non-violation content in the large model test answers and output which part of the answer violates which specific law and article.

[0163] 2) Compare with the standard clauses in the "Basic Requirements for Security of Generative Artificial Intelligence Services" and output the non-compliant items.

[0164] 3) Based on the content of 1) and 2), and the results of 6.3 and 6.4, adjust the security protection strategy of the large model and conduct model fine-tuning training.

[0165] 7. Gateway application protection module.

[0166] This module is responsible for the critical task of real-time scanning and filtering of the input and output content between the application system and the model.

[0167] As an optional implementation, the gateway application protection module includes the following sub-modules.

[0168] 7.1 Scanning Submodule.

[0169] The scanning submodule is used to perform real-time scanning of input and output content.

[0170] 7.2 Filtering Submodule.

[0171] The filtering submodule is used to filter input and output content.

[0172] 8. Safety monitoring and early warning module.

[0173] The operating status of the above functional modules is monitored in real time to identify potential security risks and issue timely alerts.

[0174] As an optional implementation, the security monitoring and early warning module includes the following sub-modules.

[0175] 8.1 Risk monitoring submodule.

[0176] The risk monitoring submodule is used to perform real-time monitoring of the MCP protection module, the RAG protection module, the API interface protection module, the security link module, the gateway application protection module, the dataset protection module, and the security audit module.

[0177] 8.2 Early Warning Submodule.

[0178] The early warning submodule is used to issue early warnings when an anomaly is detected.

[0179] 9. Security Audit Module.

[0180] This module aggregates the security logs from the aforementioned functional modules and employs various analysis methods, such as causal analysis and timeline analysis, to perform log analysis and promptly issue alerts for abnormal events. It can also remember large target models for intelligent log analysis.

[0181] As an optional implementation, the security audit module includes the following sub-modules.

[0182] 9.1 Security Log Summary and Analysis Submodule.

[0183] The security log aggregation and analysis submodule is used to aggregate security logs from various modules and analyze those security logs.

[0184] 9.2 Abnormal Event Alarm Submodule.

[0185] The abnormal event alarm submodule is used to promptly issue abnormal event alarms based on the analysis results of security logs.

[0186] This application, through the coordinated use of MCP protection modules, RAG protection modules, API interface protection modules, secure link modules, gateway application protection modules, dataset protection modules, security testing and evaluation modules, security audit modules, and security monitoring and early warning modules, can provide security protection for large-scale model applications, covering the entire lifecycle of large-scale models. These modules can work independently to provide targeted security protection, and can also collaborate with each other.

[0187] This large-scale model application management platform features targeted protection modules for multiple scenarios, ensuring strong scenario adaptability. It integrates data detection, model evaluation, and rectification guidance functions during the fine-tuning phase, forming a closed-loop management system. Security audits provide full-process tracking, real-time monitoring and early warning systems ensure timely anomaly response. The collaborative modules construct a complete protection system, effectively preventing and controlling various security risks.

[0188] The technical features of the above embodiments can be combined in any way. For the sake of brevity, not all possible combinations of the technical features in the above embodiments are described. However, as long as there is no contradiction in the combination of these technical features, they should be considered to be within the scope of this specification.

[0189] This document uses specific examples to illustrate the principles and implementation methods of this application. The descriptions of the above embodiments are only for the purpose of helping to understand the methods and core ideas of this application. Furthermore, those skilled in the art will recognize that, based on the ideas of this application, there will be changes in the specific implementation methods and application scope. Therefore, the content of this specification should not be construed as a limitation of this application.

Claims

1. A large-scale model application security management platform, characterized in that, The large-scale model application security management platform includes: MCP protection module, RAG protection module, API interface protection module, security link module, gateway application protection module, dataset protection module, security testing and evaluation module, security audit module, and security monitoring and early warning module; The MCP protection module is used to provide security protection for large models when scheduling external MCP tools. The RAG protection module is used to provide security protection for large models when performing RAG external data retrieval. The API interface protection module is used to provide security protection for large models when making API interface calls. The secure link module is used to provide security protection for large models during data transmission; The gateway application protection module is used to provide security protection for large models when conducting dialogue and question answering. The dataset protection module is used to provide data security detection when large models are being fine-tuned. The security testing and evaluation module is used to evaluate the functionality, performance, and security of the large model after training or fine-tuning. The security audit module is used to perform full-process log auditing of data interactions and operational behaviors in each module; The security monitoring and early warning module is used to monitor the MCP protection module, the RAG protection module, the API interface protection module, the security link module, the gateway application protection module, the dataset protection module, and the security audit module in real time, and issue an early warning when an anomaly is detected. The MCP protection module includes: The unified service authentication submodule is used to traverse publicly available MCP service resources based on web crawling technology, parse MCP service configuration files using Python technology, add unique representations, detailed descriptions and functional attributes to the MCP service configuration files according to the service functions, and establish an MCP service list. The MCP sandbox module is used to build a malicious behavior rule base and analyze malicious MCP services based on the malicious behavior rule base. The MCP call management submodule is used to collect the call status of each MCP service in real time based on the unique identifier of the MCP service, and record the key information of each call. The key information includes: time, initiating entity, call parameters and return result. The operation permission control submodule is used to establish a unique API key for each user based on their identity, and to construct resource access rules by combining user attributes, resource attributes and environmental conditions. At the same time, it uses encryption algorithms to encrypt permission parameters and verifies user permissions by verifying permission parameters, thus forming a dynamic permission allocation mechanism. The MCP protocol security protection submodule is used to ensure the security of the MCP protocol.

2. The large-scale model application security management platform according to claim 1, characterized in that, The RAG protection module includes: The data source security management submodule is used to build a systematic data source security management capability from two levels: trust mechanism and sensitive content identification. The classification and hierarchical control submodule is used for a document classification system based on keyword matching, regular expressions, and semantic similarity analysis. It combines multi-dimensional sensitivity levels to assign tags to documents and form metadata descriptions. The permission security optimization submodule is used to adopt a hybrid access control strategy that combines role-based access control and attribute-based access control, which enhances the flexibility and context awareness of access control while ensuring the granularity of permissions.

3. The large-scale model application security management platform according to claim 1, characterized in that, The API interface protection module includes: The multi-level authentication submodule is used to implement multi-level authentication; The traffic monitoring submodule is used to record user information, request parameters, response results, time consumption and error codes for each API call, and to handle alarms for abnormal behaviors detected according to preset alarm rules.

4. The large-scale model application security management platform according to claim 1, characterized in that, The secure link module includes: The Token Secure Link submodule is used to encrypt the data link covering the transmission of tokens through a secure transmission protocol; The call monitoring submodule is used to monitor the call status of Tokens in real time.

5. The large-scale model application security management platform according to claim 1, characterized in that, The dataset protection module includes: The dataset risk identification submodule is used to identify data poisoning, sensitive data leakage, and backdoor attacks in the dataset; The data processing submodule is used to clean and verify the identified data.

6. The large-scale model application security management platform according to claim 1, characterized in that, The security testing and evaluation module includes: The prohibited word library submodule is used to build the prohibited word library; The dictionary update submodule is used to update the prohibited dictionary; The deep-induced attack submodule is used to conduct "banned word" jailbreak tests on large models using a multi-step progressive induction strategy; the multi-step progressive induction strategy includes: seed question bank generation, multi-step induction loop, and termination output; The model security assessment submodule is used to systematically evaluate the security protection capabilities of large models in different application scenarios through static analysis, dynamic attack verification, various attack simulation strategies, and output of multi-dimensional security assessment coefficients. The various attack simulation strategies include: prohibiting sample injection, backdoor triggering test, data poisoning verification, and context-induced bypass. The multi-dimensional security assessment coefficients include: response accuracy rate, violation response rate, security blocking trigger rate, jailbreak success rate, and security risk index. The gap analysis submodule is used to benchmark against current national laws and regulations and the "Basic Requirements for Security of Generative Artificial Intelligence Services", analyze the violations and non-compliance of the large model test answers, and adjust the security protection strategy and conduct model fine-tuning training by combining the results of deep provocation attacks and security assessments.

7. The large-scale model application security management platform according to claim 1, characterized in that, The gateway application protection module includes: The scanning submodule is used to scan the input and output content in real time; The filtering submodule is used to filter input and output content.

8. The large-scale model application security management platform according to claim 1, characterized in that, The security monitoring and early warning module includes: The risk monitoring submodule is used to perform real-time monitoring of the MCP protection module, the RAG protection module, the API interface protection module, the security link module, the gateway application protection module, the dataset protection module, and the security audit module. The early warning submodule is used to issue early warnings when an anomaly is detected.

9. The large-scale model application security management platform according to claim 1, characterized in that, The security audit module includes: The security log aggregation and analysis submodule is used to aggregate the security logs from various modules and analyze the security logs. The abnormal event alarm submodule is used to promptly issue abnormal event alarms based on the analysis results of security logs.

Images