A method and device for safe management and control of generative artificial intelligence instructions

Abstract

The application discloses a generative artificial intelligence instruction safety control method and device, including the following steps: constructing a safety ontology knowledge base; creating a virtual sand table environment, which is used for caching the generated content of the generative artificial intelligence model and blocking the direct transmission of the generated content to the user end; receiving a user instruction and initializing a session context in the virtual sand table environment; inputting the user instruction into the generative artificial intelligence model to obtain generated content, time-slicing the generated content to obtain a plurality of slice contents, and performing semantic analysis on the plurality of slice contents and mapping the plurality of slice contents to the safety ontology knowledge base; based on ontology reasoning and path prediction, calculating the risk probability and logical contradiction probability of the plurality of slice contents; and executing any one of a release strategy, a blocking strategy or a correction strategy according to the risk probability and the logical contradiction probability. The application comprehensively improves the safety, accuracy and real-time performance of the generative artificial intelligence application.

Description

Technical Field

[0001] This invention relates to the field of intelligent agent technology, and specifically to a method and apparatus for the safe management of generative artificial intelligence instructions. Background Technology

[0002] To ensure the compliance and security of content generated by large language models and generative AI, enterprises and developers typically build content risk control systems. Traditional risk control systems are centered around security policy libraries, relying on predefined sensitive word lists, regular expressions, or machine learning classification models to review user input prompts and AI output results. With the development of knowledge graph technology, ontology-based knowledge bases are also being used to limit the scope of AI responses.

[0003] While existing security risk control measures can curb the spread of harmful information to some extent, they still face many challenges in scenarios involving streaming output, logical illusions, and complex semantic reasoning in generative artificial intelligence. Current technologies largely focus on input interception or post-output review. Post-output review requires waiting for complete content generation, making real-time interaction difficult. In streaming review, traditional keyword matching easily fails when sensitive content spans multiple meta-data packets, and the lack of a sandbox reasoning mechanism makes it impossible to predict subsequent risky content based on semi-generated text combined with logical rules. In streaming output scenarios, risk control detection is delayed, and generated data is directly transmitted to the user, easily leading to the leakage of the first character of sensitive content, impacting user experience and potentially causing serious information security consequences.

[0004] When existing security gateways detect risks, they typically block the session and return error messages directly, lacking automatic correction and rectification mechanisms, leading to business process interruptions. Although ontology knowledge bases can define logical constraints, existing ontology rules are mostly used as external retrieval knowledge bases and cannot form real-time constraints on the model's logical path during the generation phase.

[0005] Therefore, how to overcome the shortcomings of traditional methods with effective methods has become a pressing technical problem that needs to be solved. Summary of the Invention

[0006] In response, the present invention provides a method and apparatus for the security management of generative artificial intelligence instructions, so as to at least partially solve the above-mentioned technical problems.

[0007] This invention provides a method for security management of generative artificial intelligence instructions, comprising the following steps: S1: Construct a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels; S2: Create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; receive user instructions and initialize the session context in the virtual sandbox environment; S3: Input the user instruction into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; S4: Based on ontology reasoning and path prediction, calculate the risk probability and logical contradiction probability of several slice contents; S5: Execute any one of the following strategies based on the risk probability and the logical contradiction probability: release strategy, blocking strategy, or correction strategy.

[0008] In one aspect of this application, in step S1, each ontology node in the security ontology knowledge base is labeled with a risk level, and the relationship edges are labeled with risk transmission coefficients; wherein, the risk level includes no risk, low risk, medium risk, high risk, and extremely high risk.

[0009] One aspect of this application also includes step S6: updating the risk transmission coefficient of ontology relation edges using an online learning and batch update strategy.

[0010] In one aspect of this application, in step S3, the slice length is determined based on semantic indicators, unit length threshold, and time window threshold, and the streaming content is time-sliced ​​according to the slice length to obtain several slice contents.

[0011] In one aspect of this application, the semantic metrics include syntactic structural integrity metrics, semantic density metrics, and contextual relevance metrics.

[0012] In one aspect of this application, in step S4, the risk probability of several slice contents is calculated based on ontology path reachability analysis and temporal risk accumulation method.

[0013] In one aspect of this application, in step S4, the logical contradiction probability of several slice contents is calculated by combining direct contradiction verification, rule derivation contradiction verification, and contextual consistency verification.

[0014] In one aspect of this application, in step S5, when the risk probability is lower than the security threshold, an allow policy is executed, wherein the allow policy is to output the slice content to the user terminal; when the risk probability is higher than the security threshold, an obstruction policy is executed, wherein the obstruction policy is to terminate the generation task in the virtual sandbox and return a preset security prompt; when the probability of logical contradiction is higher than the contradiction threshold, a correction policy is executed, and the corrected slice content is output to the user terminal.

[0015] In one aspect of this application, the correction strategy includes: discarding sliced ​​content with logical deviations, generating correction prompts based on a security ontology knowledge base, and using a generative artificial intelligence model to perform corrections in conjunction with user instructions, session context, and correction prompts.

[0016] Another aspect of this application provides a generative artificial intelligence instruction security control device, comprising the following modules: The security ontology knowledge base construction module is used to build a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels. The virtual environment creation module is used to create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; it receives user instructions and initializes the session context in the virtual sandbox environment; The content mapping module is used to input the user command into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; The reasoning module is used to calculate the risk probability and logical contradiction probability of several slice contents based on ontology reasoning and path prediction. The decision control module is used to execute any one of the release strategy, blocking strategy, or correction strategy based on the risk probability and the probability of logical contradiction.

[0017] The security management method for generative artificial intelligence instructions provided in this application constructs a security ontology knowledge base, providing a rigorous and unified constraint basis for content security management. It caches generated content in a virtual sandbox environment and blocks direct transmission to the user terminal, eliminating the risk of initial character exposure in streaming output. By employing time slicing processing and ontology mapping, combined with ontology reasoning and path prediction, it calculates the probability of risks and logical contradictions, enabling early prediction of potential risks and logical illusions. Based on the calculation results, it executes release, blocking, or correction strategies, efficiently intercepting high-risk content, correcting minor logical deviations, and comprehensively improving the security, accuracy, and real-time performance of generative artificial intelligence applications. Attached Figure Description

[0018] To more clearly illustrate the technical solutions of the embodiments of the present invention, the accompanying drawings used in the embodiments will be briefly introduced below. It should be understood that the following drawings only show some embodiments of the present invention and should not be regarded as a limitation on the scope. For those skilled in the art, other related drawings can be obtained based on these drawings without creative effort.

[0019] Other features, objects, and advantages of this application will become more apparent from the following detailed description of non-limiting embodiments with reference to the accompanying drawings: Figure 1 This is a schematic diagram of the structure of the security management device for generative artificial intelligence instructions provided in an embodiment of the present invention; Figure 2 This is a schematic diagram of the security management method for generative artificial intelligence instructions provided in an embodiment of the present invention. Detailed Implementation

[0020] The present invention will now be described in detail with reference to the specific embodiments shown in the accompanying drawings. However, these embodiments do not limit the present invention, and any structural, methodological, or functional modifications made by those skilled in the art based on these embodiments are included within the scope of protection of the present invention.

[0021] The scope of the embodiments described herein includes the entire scope of the claims and all available equivalents thereof. Throughout this document, the terms “first,” “second,” etc., are used only to distinguish one element from another without requiring or implying any actual relationship or order between the elements. Indeed, a first element can also be referred to as a second element, and vice versa. Furthermore, the terms “comprising,” “including,” or any other variations thereof are intended to cover non-exclusive inclusion, such that a structure, apparatus, or device that comprises a list of elements includes not only those elements but also other elements not expressly listed, or elements inherent to such a structure, apparatus, or device. Without further limitations, an element defined by the phrase “comprising one…” does not exclude the presence of other identical elements in the structure, apparatus, or device that includes said element. The various embodiments described herein are presented in a progressive manner, with each embodiment focusing on its differences from other embodiments. Similar or identical parts between embodiments can be referred to interchangeably.

[0022] The terms "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer" used in this document to indicate orientation or positional relationships are based on the orientation or positional relationships shown in the accompanying drawings and are used only for the convenience of describing this document and simplifying the description. They do not indicate or imply that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation, and therefore should not be construed as limiting the invention. In the description herein, unless otherwise specified and limited, the terms "installed," "connected," and "linked" should be interpreted broadly. For example, they can refer to mechanical or electrical connections, or internal connections between two elements, or direct connections or indirect connections through an intermediate medium. Those skilled in the art can understand the specific meaning of the above terms according to the specific circumstances.

[0023] This application proposes a method for security management of generative artificial intelligence instructions, including the following steps: Step S1: Construct a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels. Each ontology node in the security ontology knowledge base is labeled with a risk level, and the relationship edges are labeled with risk transmission coefficients. The risk levels include no risk, low risk, medium risk, high risk, and extremely high risk.

[0024] Here, the security ontology knowledge base uses OWL2 (Web Ontology Language 2) as the ontology description language and combines it with the RDF triple model to store entity relationships. Specifically, it includes a top-level ontology, a domain ontology, and an application ontology. The top-level ontology defines general security concepts such as risk, compliance, and violation. The domain ontology defines professional concepts and hierarchical relationships for specific industries such as healthcare, law, and finance. The application ontology contains specific business rules and instance data. The security ontology knowledge base uses a combination of expert annotation and semi-automatic extraction to acquire domain knowledge. Core rules are annotated by domain experts, while auxiliary knowledge is semi-automatically extracted from the domain corpus using natural language processing tools and then manually reviewed before being added to the database. The security ontology knowledge base supports an incremental update mechanism, generating a new version snapshot with each update. Semantic version numbers are used for management to achieve traceability and rollback.

[0025] Step S2: Create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; receive user instructions and initialize the session context in the virtual sandbox environment; In practice, the virtual sandbox environment is located between the generative AI model server and the user terminal transmission link. It caches the generated content of the generative AI model's streaming output, preventing the generated content from being directly transmitted to the user terminal without security checks, thus achieving physical isolation and data interception between the generated content and the user terminal. It receives user commands sent from the user terminal and transmits them completely into the virtual sandbox environment, where the initialization operation of the generative AI model's session context is completed. The initialization process loads the model's historical session information, user identity identifier, and current task configuration parameters, constructing a running state consistent with the real business scenario.

[0026] Step S3: Input the user instruction into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; specifically, determine the slice length according to semantic indicators, unit length threshold and time window threshold, and perform time slicing on the streaming generated content according to the slice length to obtain several slice content.

[0027] Here, the semantic metrics include syntactic structure completeness, semantic density, and contextual relevance. The syntactic structure completeness metric is based on punctuation and phrase ending markers. Sentence-ending punctuation, sentence-internal punctuation, and suggestive punctuation have different weights. When the sum of accumulated punctuation weights in the cached content reaches a first set threshold, the current content is considered to have a complete syntactic structure. The semantic density metric uses a lightweight named entity recognition module to count the number of entities and relational trigger words in the current cached content, and calculates a comprehensive semantic density value according to set weight coefficients. When the semantic density value reaches a second set threshold, the current content is considered to possess the semantic information required for ontology mapping and reasoning. The contextual relevance metric calculates the cosine similarity between the current cached content and the previous slice summary using a lightweight sentence vector model. Semantic continuity is judged based on the cosine similarity value. When the value is below a third set threshold, it is considered a topic jump; when the value is above the third set threshold, it is considered semantic continuation.

[0028] In practice, when the number of tokens in the buffer reaches its limit, slicing is performed regardless of the semantic indicator status. Standard slicing is executed when both syntactic completeness and semantic density meet the set conditions, ensuring the buffer content has a complete syntactic structure and sufficient semantic information. When syntactic completeness meets the standard but semantic density does not reach the second set threshold, tokens are continuously accumulated until the semantic density meets the condition or the maximum number of tokens is reached. When the context relevance is lower than the third set threshold, a topic jump is detected, and slicing is forcibly executed. Slice length and time parameters are set according to the specification range. The minimum slice length ranges from 5 to 15 tokens, with a default value of 8 tokens, which forms effective semantic units without increasing computational burden. The maximum slice length ranges from 30 to 80 tokens, with a default value of 50 tokens, balancing semantic information completeness with real-time ontology mapping processing. The upper limit of the time window ranges from 30 milliseconds to 100 milliseconds, with a default value of 50 milliseconds. Preferably, when the current slice deduction result is at a medium risk level, the maximum slice length is reduced to improve the granularity of review; after multiple consecutive low-risk slices, the default parameters are restored. The model's word output rate is monitored in real time; when the rate exceeds a fourth set threshold, the upper limit of the time window is shortened to avoid data backlog; when the rate is below the fourth set threshold, the upper limit of the time window is extended to accumulate sufficient semantic information. In highly sensitive business scenarios, a smaller maximum slice length and a lower semantic density threshold are used to achieve more refined review control.

[0029] Step S4: Based on ontology reasoning and path prediction, calculate the risk probability and logical contradiction probability of several slice contents; specifically, calculate the risk probability of several slice contents based on ontology path reachability analysis and temporal risk accumulation method. Combine direct contradiction verification, rule derivation contradiction verification, and contextual consistency verification to calculate the logical contradiction probability of several slice contents.

[0030] In practice, the calculation of the risk probability specifically includes: first, extracting relevant entities from the slice content; calculating the weighted shortest path distance from each entity to the risk node in the ontology knowledge graph; obtaining the path risk score of a single entity in an exponential decay manner; then calculating the slice-level path risk score by combining the relationship between entities; and finally, accumulating the risk in the time-series dimension by integrating the historical evaluation results of multiple consecutive slice content through a Bayesian temporal update mechanism, and finally obtaining the risk probability of the slice content.

[0031] The calculation of the logical contradiction probability specifically includes: First, a direct contradiction check is performed to determine whether the statements in the slice content directly conflict with the facts recorded in the ontology knowledge base. Next, a rule-derived contradiction check is performed, using inference rules such as transitivity, symmetry, and disjointness constraints included in the ontology to derive implicit knowledge and determine whether the slice content conflicts with it. Then, a contextual consistency check is performed to determine whether the statements in the slice contradict the preceding generated content within the same session. Based on the results of these checks, the logical contradiction probability of the slice content is calculated in parallel.

[0032] Step S5: Execute any one of the following strategies based on the risk probability and the logical contradiction probability: a release strategy, a blocking strategy, or a correction strategy. Specifically, when the risk probability is lower than the security threshold, the release strategy is executed, which involves outputting the slice content to the user terminal; when the risk probability is higher than the security threshold, the blocking strategy is executed, which involves terminating the generation task within the virtual sandbox and returning a preset security prompt; when the logical contradiction probability is higher than the contradiction threshold, the correction strategy is executed, and the corrected slice content is output to the user terminal.

[0033] Here, the correction strategy includes: discarding sliced ​​content with logical deviations, generating correction prompts based on a security ontology knowledge base, and using a generative artificial intelligence model to make corrections in conjunction with user instructions, session context, and correction prompts.

[0034] The present invention also includes step S6: updating the risk transmission coefficient of the ontology relation edges using an online learning and batch update strategy.

[0035] In practice, each completed correction or blocking case is recorded, and a multi-dimensional feature vector, including text features, entity features, path features, decision features, and outcome features, is extracted for model updates. In online learning mode, updates are performed immediately after each correction or blocking event, using an exponentially weighted moving average method to adjust the risk transmission coefficient of the corresponding ontology relation edges, while limiting the magnitude of single parameter changes to avoid parameter drift caused by anomalous data. In batch update mode, the accumulated case library is trained centrally at fixed intervals, and a gradient boosting tree model is used to optimize the relevant parameters for calculating risk probability. Before updating, constraints on the number of samples and the ratio of positive to negative samples are met, and all parameters are set with reasonable value boundaries. Simultaneously, recent multi-round parameter snapshots are retained; if the model performance significantly declines after an update, it is automatically rolled back to a historical stable version.

[0036] The present invention also provides a generative artificial intelligence instruction security control device, comprising the following modules: The security ontology knowledge base construction module 201 is used to construct a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels. The virtual environment creation module 202 is used to create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; it receives user instructions and initializes the session context in the virtual sandbox environment; The content mapping module 203 is used to input the user instruction into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; The reasoning module 204 is used to calculate the risk probability and logical contradiction probability of several slice contents based on ontology reasoning and path prediction. The decision control module 205 is used to execute any one of the release strategy, blocking strategy, or correction strategy based on the risk probability and the probability of logical contradiction.

[0037] This invention can be an apparatus, method, and / or computer program product. A computer program product may include a readable storage medium having computer-readable program instructions loaded thereon for causing a processor to implement various aspects of the invention.

[0038] Storage media can be tangible devices that hold and store instructions for use by instruction execution devices. Storage media can include, for example, but not limited to, electrical storage devices, magnetic storage devices, optical storage devices, electromagnetic storage devices, semiconductor storage devices, or any suitable combination thereof. More specific examples (a non-exhaustive list) of readable storage media include: portable computer disks, hard disks, random access memory (RAM), read-only memory (ROM), erasable programmable read-only memory (EPROM or flash memory), static random access memory (SRAM), portable compact disc read-only memory (CD-ROM), digital multifunction disc (DVD), memory sticks, floppy disks, mechanical encoding devices, such as punch cards or recessed protrusions storing instructions thereon, and any suitable combination thereof.

[0039] It should be understood that although this specification describes embodiments, not every embodiment contains only one independent technical solution. This way of describing the specification is only for clarity. Those skilled in the art should regard the specification as a whole. The technical solutions in each embodiment can also be appropriately combined to form other embodiments that can be understood by those skilled in the art.

[0040] The detailed descriptions listed above are merely specific descriptions of feasible embodiments of the present invention, and are not intended to limit the scope of protection of the present invention. All equivalent embodiments or modifications made without departing from the spirit of the present invention should be included within the scope of protection of the present invention.

Claims

1. A method for security management of generative artificial intelligence instructions, characterized in that, Includes the following steps: S1: Construct a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels; S2: Create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; receive user instructions and initialize the session context in the virtual sandbox environment; S3: Input the user instruction into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; S4: Based on ontology reasoning and path prediction, calculate the risk probability and logical contradiction probability of several slice contents; S5: Execute any one of the following strategies based on the risk probability and the logical contradiction probability: release strategy, blocking strategy, or correction strategy.

2. The generative artificial intelligence instruction security management method according to claim 1, characterized in that, In step S1, each ontology node in the security ontology knowledge base is labeled with a risk level, and the relationship edges are labeled with risk transmission coefficients; wherein, the risk levels include no risk, low risk, medium risk, high risk, and extremely high risk.

3. The generative artificial intelligence instruction security management method according to claim 2, characterized in that, It also includes step S6: updating the risk transmission coefficient of ontology relation edges using an online learning and batch update strategy.

4. The generative artificial intelligence instruction security management method according to claim 1, characterized in that, In step S3, the slice length is determined based on semantic indicators, unit length threshold, and time window threshold, and the streaming content is time-sliced ​​according to the slice length to obtain several slice contents.

5. The generative artificial intelligence instruction security management method according to claim 4, characterized in that, The semantic metrics include syntactic structure completeness metrics, semantic density metrics, and contextual relevance metrics.

6. The generative artificial intelligence instruction security management method according to claim 1, characterized in that, In step S4, the risk probability of several slice contents is calculated based on ontology path reachability analysis and temporal risk accumulation method.

7. The generative artificial intelligence instruction security management method according to claim 1, characterized in that, In step S4, the logical contradiction probability of several slice contents is calculated by combining direct contradiction verification, rule derivation contradiction verification, and contextual consistency verification.

8. The generative artificial intelligence instruction security management method according to claim 1, characterized in that, In step S5, when the risk probability is lower than the security threshold, an allow policy is executed, which means outputting the slice content to the user terminal; when the risk probability is higher than the security threshold, a blocking policy is executed, which means terminating the generation task in the virtual sandbox and returning a preset security prompt; when the probability of logical contradiction is higher than the contradiction threshold, a correction policy is executed, and the corrected slice content is output to the user terminal.

9. The generative artificial intelligence instruction security management method according to claim 8, characterized in that, The correction strategy includes: discarding sliced ​​content with logical deviations, generating correction prompts based on a security ontology knowledge base, and using a generative artificial intelligence model to correct the content by combining user instructions, session context, and correction prompts.

10. A generative artificial intelligence instruction security control device, characterized in that, Includes the following modules: The security ontology knowledge base construction module is used to build a security ontology knowledge base, which includes security compliance rules, business logic constraints, entity relationship graphs, and risk levels. The virtual environment creation module is used to create a virtual sandbox environment, which is used to cache the generated content of the generative artificial intelligence model and block the direct transmission of the generated content to the user terminal; it receives user instructions and initializes the session context in the virtual sandbox environment; The content mapping module is used to input the user command into the generative artificial intelligence model to obtain generated content, perform time slicing on the generated content to obtain several slice content, perform semantic parsing on the several slice content and map it to the security ontology knowledge base; The reasoning module is used to calculate the risk probability and logical contradiction probability of several slice contents based on ontology reasoning and path prediction. The decision control module is used to execute any one of the release strategy, blocking strategy, or correction strategy based on the risk probability and the probability of logical contradiction.

Images