Method and apparatus for monitoring artificial intelligence system, electronic device and storage medium

By constructing a compliance constraint topology graph model and a comprehensive security scoring mechanism, the shortcomings of artificial intelligence system monitoring methods in terms of accuracy, real-time performance, and interpretability are addressed. This enables deep semantic understanding and risk assessment of user interaction content, meeting enterprise-level security compliance requirements.

CN123114907APending Publication Date: 2026-11-17GUANGZHOU ZHONGCHENG QINGYUN CIYUAN TECHNOLOGY CO LTD
View PDF 0 Cites 0 Cited by

Patent Information

Application Number
CN202611136591.4
Authority / Receiving Office
CN · China
Patent Type
Applications(China)
Current Assignee / Owner
Filing Date
2026-07-29
Publication Date
2026-11-17

AI Technical Summary

Technical Problem

Existing AI-based monitoring methods fall short in terms of accuracy, real-time performance, deployment costs, and interpretability, failing to effectively meet the stringent security and compliance requirements of enterprise applications.

Method used

By constructing a compliance constraint topology graph model, combining semantic vectors to calculate topological density field strength and manifold hedging scores, and combining temporal evolution feature information to conduct a comprehensive security score, we can achieve deep semantic understanding and risk assessment of user interaction content.

Benefits of technology

It improves the accuracy and interpretability of AI system monitoring, reduces deployment costs, and meets the security and compliance requirements of enterprise applications.

✦ Generated by Eureka AI based on patent content.

Smart Images

  • Figure CN123114907A_ABST
    Figure CN123114907A_ABST
Patent Text Reader

Abstract

The application belongs to the technical field of artificial intelligence, and discloses a kind of monitoring method, device, electronic equipment and storage medium of artificial intelligence system, the method comprises: obtaining the user interaction content information of artificial intelligence system, according to user interaction content information, combine compliance constraint topological graph model, carry out topological density field intensity calculation, obtain the topological density field intensity information of user interaction content information, based on topological density field intensity information, combine compliance violation balance parameter information and carry out manifold hedging score calculation, obtain the manifold hedging score of user interaction content information, based on manifold hedging score, combine the time sequence evolution characteristic information of user interaction content information and carry out comprehensive score processing, obtain the comprehensive security score of user interaction content information, based on comprehensive security score, to artificial intelligence system carries out monitoring processing, obtains the monitoring result of artificial intelligence system;Through the above method, the monitoring precision of artificial intelligence system is improved.
Need to check novelty before this filing date? Find Prior Art

Description

Technical Field

[0001] This application belongs to the technical field of artificial intelligence, and more specifically, relates to a monitoring method, device, electronic device and storage medium for an artificial intelligence system. Background Technology

[0002] With the widespread deployment of large language models in enterprise-level scenarios such as finance, government affairs, and customer service, the security and compliance management of the input and output content of artificial intelligence systems has become a rigid requirement in the industry. Existing technologies mainly suffer from two types of problems: one type of method, while leveraging the semantic understanding advantages of large-scale language models and modeling content security detection as a conditional generation task, can achieve high detection accuracy on standard test sets. However, it relies on high-performance GPUs for inference, resulting in a single detection latency that is typically above hundreds of milliseconds, making deployment costs high and response speeds difficult to meet the requirements of real-time enterprise-level business.

[0003] Another type of method employs lightweight shallow classifiers or feature matching strategies. While inference can be completed on CPUs with low latency, it generally suffers from a high false positive rate for normal behavior. Specifically, a large amount of compliant content is incorrectly labeled as risky behavior, requiring significant manpower from operations and maintenance personnel to review alerts, which severely undermines the practical effectiveness of automated security protection. Furthermore, most existing methods are end-to-end black-box models, making it difficult to provide traceable and auditable mathematical explanations when security incidents occur. This poses a significant obstacle for enterprises in passing security compliance reviews. Therefore, existing AI system security monitoring methods are insufficient in terms of accuracy, real-time performance, deployment cost, and interpretability, failing to effectively meet the stringent security compliance requirements of enterprise applications for AI systems.

[0004] To address the aforementioned issues, existing technologies urgently need improvement. Summary of the Invention

[0005] The purpose of this application is to provide a monitoring method, device, electronic device, and storage medium for artificial intelligence systems, in order to address the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise-level applications for artificial intelligence systems. Through a multi-dimensional scoring mechanism, it achieves deep semantic understanding and risk assessment of user interaction content, thereby improving the monitoring accuracy of artificial intelligence systems.

[0006] Firstly, this application provides a monitoring method for an artificial intelligence system, including: To obtain user interaction information from artificial intelligence systems; Based on the user interaction content information, and combined with the corresponding preset compliance constraint topology graph model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information. Based on the topological density field strength information, and combined with the preset compliance and non-compliance balance parameter information, the manifold hedging score is calculated to obtain the manifold hedging score corresponding to the user interaction content information. Based on the manifold hedging score, and combined with the temporal evolution feature information corresponding to the user interaction content information, a comprehensive scoring process is performed to obtain the comprehensive security score corresponding to the user interaction content information. Based on the comprehensive security score, the artificial intelligence system is monitored to obtain the monitoring results of the artificial intelligence system.

[0007] The monitoring method for artificial intelligence systems provided in this application enables the monitoring of AI systems. After acquiring user interaction content information of the AI ​​system, it calculates a manifold hedging score using a preset compliance constraint topology graph model and preset compliance / violation balance parameters. This score is then combined with the temporal evolution characteristics of the user interaction content information to perform comprehensive scoring, resulting in a comprehensive security score. This comprehensive security score is used to monitor the AI ​​system, addressing the shortcomings of existing AI system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise applications for AI systems. Through a multi-dimensional scoring mechanism, it achieves deep semantic understanding and risk assessment of user interaction content, thereby improving the monitoring accuracy of AI systems.

[0008] Optionally, the topology density field strength includes compliant topology density field strength and non-compliant topology density field strength; based on the user interaction content information and in conjunction with the corresponding preset compliant constraint topology graph model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information, including: Obtain a pre-defined compliance constraint topology graph model corresponding to the artificial intelligence system; the compliance constraint topology graph model is a pre-constructed topology graph model with each compliance semantic in a pre-defined set of compliance semantics and each violation semantic in a pre-defined set of violation semantics as nodes, and pre-defined compliance associations and pre-defined violation associations between each compliance semantic and each violation semantic as edges. Each node is configured with a compliance semantic vector corresponding to the compliance semantic and a violation semantic vector corresponding to the violation semantic. The edges are then assigned values ​​based on pre-defined compliance quantification values ​​corresponding to the compliance associations and pre-defined violation quantification values ​​corresponding to the violation associations. Convert the user interaction content information into a semantic vector; The semantic vector is input into the compliance constraint topology graph model for node filtering to obtain the compliance manifold nodes and violation manifold nodes corresponding to the semantic vector; Based on the compliant semantic vector of the compliant manifold node and the non-compliant semantic vector of the non-compliant manifold node, and combined with the preset semantic compliance density distribution coefficient and semantic non-compliant density distribution coefficient, the similarity of the semantic vectors is calculated to obtain the compliant topological density field intensity and the non-compliant topological density field intensity corresponding to the semantic vectors.

[0009] The monitoring method for artificial intelligence systems provided in this application can monitor artificial intelligence systems. By constructing a refined compliance constraint topology graph model and combining it with semantic vectors to calculate the topology density field strength, it can more accurately identify compliance and violation tendencies in user interaction content, thereby improving the accuracy and interpretability of detection.

[0010] Optionally, after converting the user interaction content information into a semantic vector, the method further includes: The average semantic vectors of all users are used to calculate the group semantic mean of all users. The difference between the semantic vector of each user and the semantic mean of the group is calculated to obtain the individual deviation degree of each user; The individual deviation is compared with a preset deviation threshold; When the individual deviation is greater than a preset deviation threshold, the user corresponding to the individual deviation is identified as a user subject to detailed investigation. When the individual deviation is less than or equal to a preset deviation threshold, the user corresponding to the individual deviation is identified as a coarse-screened user. Different monitoring strategies are adopted for the detailed investigation users and the coarse screening users to monitor the artificial intelligence system of the detailed investigation users and the coarse screening users.

[0011] Optionally, the temporal evolution feature information includes a semantic long-term monitoring score, a setting offset detection score, and a malicious premeditation detection score; based on the manifold hedging score, combined with the temporal evolution feature information corresponding to the user interaction content information, a comprehensive scoring process is performed to obtain a comprehensive security score corresponding to the user interaction content information, including: The semantic vector corresponding to the user interaction content information is subjected to historical monitoring and scoring processing to obtain the long-term semantic monitoring score and the set offset detection score of the user interaction content information. Malicious premeditation detection processing is performed on the semantic vector corresponding to the user interaction content information to obtain the malicious premeditation detection score of the user interaction content information; Based on the manifold hedging score, the semantic long-term monitoring score, the set offset detection score, and the malicious premeditation detection score, a weighted summation is performed to obtain the comprehensive security score of the user interaction content information.

[0012] The monitoring method for artificial intelligence systems provided in this application can monitor artificial intelligence systems. By incorporating time-series evolution characteristic information into a comprehensive scoring system, it can more comprehensively assess the security risks of user interaction content, effectively identify long-term trends and potential malicious behaviors, and improve the predictability and accuracy of monitoring.

[0013] Optionally, the semantic vector corresponding to the user interaction content information is subjected to historical monitoring and scoring processing to obtain a long-term semantic monitoring score and a set offset detection score for the user interaction content information, including: Based on the preset short-term decay coefficient and the preset long-term decay coefficient, the short-term memory state and long-term memory state are calculated on the semantic vector corresponding to the user interaction content information to obtain the short-term memory state parameters and long-term memory state parameters of the user interaction content information. Based on the short-term memory state parameters, the risk index is calculated by combining the Laplace matrix of the compliance constraint topology graph model to obtain the short-term risk index. Based on the long-term memory state parameters, the risk index is calculated by combining the Laplace matrix of the compliance constraint topology graph model to obtain the long-term risk index. The maximum value among the short-term memory state parameters and the long-term memory state parameters is determined as the semantic long-term monitoring score of the user interaction content information; Based on the long-term memory state parameters and the preset baseline state vector, the difference is calculated to obtain the set offset detection score of the user interaction content information.

[0014] Optionally, malicious premeditation detection processing is performed on the semantic vector corresponding to the user interaction content information to obtain a malicious premeditation detection score for the user interaction content information, including: According to the preset sensitive topic classification rules, the semantic vector corresponding to the user interaction content information is subjected to sensitive topic feature extraction to obtain the sensitive topic feature vector of the artificial intelligence system, and the number of times the sensitive topic appears in the sensitive topic feature vector is determined. According to the preset user rejection intent recognition rules, rejection features are extracted from the semantic vector corresponding to the user interaction content information to obtain the user's rejection semantic feature vector, and the number of times the rejection semantic appears in the rejection semantic feature vector is determined. Based on the rejection semantic feature vector, the sensitive topic feature vector, the number of times the sensitive topic appears, and the number of times the rejection semantic appears, active sensitive topic statistics and repeated mention counts are performed to obtain the number of times the artificial intelligence system actively mentions sensitive topics, and the number of times the artificial intelligence system mentions any sensitive topic again after the user rejects any sensitive topic. The malicious intent detection score of the user interaction content information is obtained by calculating the ratio of the number of times the sensitive topic is actively mentioned to the number of times the sensitive topic is repeatedly mentioned.

[0015] Optionally, based on the comprehensive security score, the artificial intelligence system is monitored to obtain monitoring results for the artificial intelligence system, including: The system determines whether the comprehensive security score is greater than or equal to a preset security control threshold, whether the manifold hedging score is greater than or equal to a preset semantic violation compliance threshold, whether the semantic long-term monitoring score is greater than or equal to a preset long-term monitoring threshold, whether the set offset detection score is greater than or equal to a preset set offset threshold, and whether the malicious premeditation detection score is greater than or equal to a preset malicious premeditation threshold. When the comprehensive security score is greater than or equal to the security control threshold, or the manifold hedging score is greater than or equal to the semantic violation compliance threshold, or the semantic long-term monitoring score is greater than or equal to the long-term monitoring threshold, or the set offset detection score is greater than or equal to the set offset threshold, or the malicious premeditation detection score is greater than or equal to the malicious premeditation threshold, the artificial intelligence system is intervened and controlled according to the preset intervention strategy to obtain the monitoring results of the artificial intelligence system; When the overall security score is less than the security control threshold, the manifold hedging score is less than the semantic violation compliance threshold, the semantic long-term monitoring score is less than the long-term monitoring threshold, the set offset detection score is less than the set offset threshold, and the malicious premeditation detection score is less than the malicious premeditation threshold, it is determined that no intervention or control is required for the artificial intelligence system, and the monitoring result of the artificial intelligence system is obtained.

[0016] Secondly, this application provides a monitoring device for an artificial intelligence system, comprising: The acquisition module is used to acquire user interaction content information from the artificial intelligence system. The first calculation module is used to calculate the topological density field strength based on the user interaction content information and the corresponding preset compliance constraint topological graph model of the artificial intelligence system, so as to obtain the topological density field strength information corresponding to the user interaction content information. The second calculation module is used to calculate the manifold hedging score based on the topological density field strength information and the preset compliance and non-compliance balance parameter information, so as to obtain the manifold hedging score corresponding to the user interaction content information. The comprehensive scoring module is used to perform comprehensive scoring processing based on the manifold hedging score and the temporal evolution feature information corresponding to the user interaction content information to obtain the comprehensive security score corresponding to the user interaction content information. The monitoring module is used to monitor the artificial intelligence system based on the comprehensive security score and obtain the monitoring results of the artificial intelligence system.

[0017] The monitoring device of this artificial intelligence system calculates a manifold hedging score using a pre-set compliance constraint topology graph model and pre-set compliance / violation balance parameter information. This score is then combined with the temporal evolution characteristics of user interaction content to perform comprehensive scoring and obtain a comprehensive security score. Based on this comprehensive security score, the system is monitored. This addresses the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise applications for artificial intelligence systems. Through a multi-dimensional scoring mechanism, the system achieves deep semantic understanding and risk assessment of user interaction content, thereby improving the monitoring accuracy of the artificial intelligence system.

[0018] Thirdly, this application provides an electronic device including a processor and a memory, the memory storing a computer program executable by the processor, wherein when the processor executes the computer program, it performs the steps in the monitoring method of the artificial intelligence system described above.

[0019] Fourthly, this application provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, performs the steps of the monitoring method of the artificial intelligence system described above.

[0020] Beneficial Effects: The monitoring method, device, electronic equipment, and storage medium for artificial intelligence systems provided in this application calculate a manifold hedging score using a preset compliance constraint topology graph model and preset compliance / violation balance parameter information. This score is then combined with the temporal evolution characteristics of user interaction content information to perform comprehensive scoring processing, resulting in a comprehensive security score. This comprehensive security score is used to monitor the artificial intelligence system, addressing the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise-level applications for artificial intelligence systems. Through a multi-dimensional scoring mechanism, the method achieves deep semantic understanding and risk assessment of user interaction content, thereby improving the monitoring accuracy of the artificial intelligence system. Attached Figure Description

[0021] Figure 1 A flowchart of a monitoring method for an artificial intelligence system provided in an embodiment of this application.

[0022] Figure 2 This is a schematic diagram of the structure of the monitoring device for the artificial intelligence system provided in the embodiments of this application.

[0023] Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application.

[0024] Labeling Explanation: 1. Acquisition Module; 2. First Calculation Module; 3. Second Calculation Module; 4. Comprehensive Scoring Module; 5. Monitoring Module; 301. Processor; 302. Memory; 303. Communication Bus. Detailed Implementation

[0025] The technical solutions of the embodiments of this application will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of the embodiments. The components of the embodiments of this application described and shown in the accompanying drawings can generally be arranged and designed in various different configurations. Therefore, the following detailed description of the embodiments of this application provided in the accompanying drawings is not intended to limit the scope of the claimed application, but merely represents selected embodiments of this application. All other embodiments obtained by those skilled in the art based on the embodiments of this application without inventive effort are within the scope of protection of this application.

[0026] It should be noted that similar reference numerals and letters in the following figures indicate similar items; therefore, once an item is defined in one figure, it does not need to be further defined and explained in subsequent figures. Furthermore, in the description of this application, terms such as "first," "second," etc., are used only to distinguish descriptions and should not be construed as indicating or implying relative importance.

[0027] Please refer to Figure 1 , Figure 1 This application discloses a method for monitoring an artificial intelligence system, as described in some embodiments, for monitoring the artificial intelligence system, comprising the following steps: Step S101: Obtain user interaction content information from the artificial intelligence system; Step S102: Based on the user interaction content information and combined with the corresponding preset compliance constraint topology map model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information. Step S103: Based on the topological density field strength information and combined with the preset compliance and non-compliance balance parameter information, the manifold hedging score is calculated to obtain the manifold hedging score corresponding to the user interaction content information. Step S104: Based on the manifold hedging score, a comprehensive score is obtained by combining the temporal evolution feature information corresponding to the user interaction content information with the comprehensive security score corresponding to the user interaction content information. Step S105: Based on the comprehensive security score, monitor the artificial intelligence system to obtain the monitoring results of the artificial intelligence system.

[0028] This monitoring method for an artificial intelligence system, after acquiring user interaction content information, calculates a manifold hedging score using a pre-set compliance constraint topology graph model and pre-set compliance / violation balance parameters. This score is then combined with the temporal evolution characteristics of the user interaction content to obtain a comprehensive security score. This comprehensive security score is used to monitor the artificial intelligence system, thus addressing the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise applications. Through a multi-dimensional scoring mechanism, it achieves deep semantic understanding and risk assessment of user interaction content, improving the monitoring accuracy of the artificial intelligence system.

[0029] Specifically, in step S101, the dialogue text between the user and artificial intelligence systems such as chatbots and intelligent customer service is obtained, thus acquiring user interaction content information of the artificial intelligence system. This user interaction content information refers to all data generated during the interaction between the user and the artificial intelligence system, including text, voice, and images. This data carries the intent and behavioral information of both the user and the artificial intelligence system.

[0030] Specifically, the topology density field strength includes compliant topology density field strength and non-compliant topology density field strength. Optionally, in step S102, based on the user interaction content information and combined with the corresponding preset compliant constraint topology graph model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information, specifically including: Obtain the pre-defined compliance constraint topology graph model corresponding to the artificial intelligence system; the compliance constraint topology graph model is constructed in advance by taking each compliance semantic in the pre-defined compliance semantic set and each violation semantic in the pre-defined violation semantic set as nodes, taking the pre-defined compliance association relationship between each compliance semantic and the pre-defined violation association relationship between each violation semantic as edges, configuring the compliance semantic vector corresponding to the compliance semantic and the violation semantic vector corresponding to the violation semantic for each node, and assigning values ​​to each edge by the pre-defined compliance quantification assignment corresponding to the compliance association relationship and the pre-defined violation quantification assignment corresponding to the violation association relationship; Convert user interaction content information into semantic vectors; The semantic vectors are input into the compliance constraint topology graph model for node filtering, resulting in the compliance manifold nodes and violation manifold nodes corresponding to the semantic vectors. Based on the compliant semantic vector of the compliant manifold node and the non-compliant semantic vector of the non-compliant manifold node, and combined with the preset semantic compliance density distribution coefficient and semantic non-compliant density distribution coefficient, the similarity of the semantic vectors is calculated to obtain the compliant topological density field intensity and the non-compliant topological density field intensity corresponding to the semantic vector.

[0031] It should be noted that the topology density field strength is subdivided into compliant topology density field strength and non-compliant topology density field strength, aiming to quantitatively evaluate user interaction content information from two dimensions: compliance and non-compliance. Specifically, the compliant topology density field strength measures the degree to which user interaction content information conforms to compliant semantics, while the non-compliant topology density field strength measures its degree to which it conforms to non-compliant semantics.

[0032] In step S102, a pre-constructed knowledge graph, namely a compliance constraint topology graph model, is obtained. Its construction process includes: using each compliance semantic in a pre-defined set of compliance semantics and each violation semantic in a pre-defined set of violation semantics as nodes of the graph; using pre-defined compliance relationships and pre-defined violation relationships between compliance semantics as edges of the graph; configuring compliance semantic vectors and violation semantic vectors for each node; and assigning values ​​to each edge based on pre-defined compliance quantification values ​​for compliance relationships and pre-defined violation quantification values ​​for violation relationships. Thus, this topology graph model (i.e., the compliance constraint topology graph model) can comprehensively represent compliance and violation semantics and their interrelationships, providing a structured knowledge foundation for subsequent density field intensity calculations. The compliance semantic set refers to a pre-determined set of semantics representing compliance, using rules that identify legal, positive, and constructive content, such as "normal consultation" and "product introduction." The violation semantic set refers to a pre-determined set of semantics representing violation, using rules that identify illegal, negative, and harmful content, such as "privacy leak" and "sensitive political speech." Furthermore, the pre-defined compliance relationships and their corresponding compliance quantifications, as well as the pre-defined violation relationships and their corresponding violation quantifications, can be obtained based on the association criteria and scoring criteria set in patent opinions and related books.

[0033] In practical implementation, existing technologies such as natural language processing can be used. For example, pre-trained embedding models (such as Word2Vec and GloVe) or pre-defined contextual embedding models (such as BERT and GPT series models) can be used to encode the text content, mapping each word or sentence to a point in a high-dimensional vector space. This converts the user interaction content information into a numerical vector that can represent its semantic features, i.e., a semantic vector. This vectorized representation enables computers to understand and process the deeper meaning of the text.

[0034] For example, the semantic vector is calculated as follows: ; in, Let be the semantic vector corresponding to the user interaction content information at time t; This refers to user interaction information at time t. This is an embedding model (one of the ways to calculate semantic vectors, which can be adjusted according to actual needs).

[0035] Optionally, in step S102, after converting the user interaction content information into a semantic vector, the method further includes: The group semantic mean of all users is calculated based on the semantic vectors of all users. The difference between the semantic vector of each user and the group semantic mean is calculated to obtain the individual deviation of each user. Compare the individual deviation with a preset deviation threshold; When an individual's deviation exceeds a preset deviation threshold, the user corresponding to that individual's deviation will be identified as a user subject to detailed investigation. When an individual's deviation is less than or equal to a preset deviation threshold, the user corresponding to that individual's deviation will be identified as a coarse-screened user. Different monitoring strategies are adopted for users who perform detailed checks and users who perform coarse checks, and the artificial intelligence system is used to monitor these users.

[0036] It should be noted that the group semantic mean is the average or median of the semantic vectors of all user interactions within a specific time period or across the entire user group. This mean represents the overall semantic behavior pattern or "normal" behavior baseline of the user group, and can be calculated by aggregating the semantic vectors of all users and taking the average, median, or other statistical methods. Individual deviation refers to the distance or degree of difference between a single user's semantic vector and the group semantic mean. This deviation can be calculated using various distance metrics, such as Euclidean distance, cosine similarity (inverted or converted to distance), Mahalanobis distance, etc., to quantify the degree of deviation of user behavior from the group's norm.

[0037] Specifically, in step S102, after obtaining the semantic vectors, the average semantic value of all users can be obtained by averaging the semantic vectors of all users. This average semantic value represents the average semantic characteristics of all current user interaction behaviors and can be regarded as a baseline of "normal" or "typical" user behavior.

[0038] The specific method for calculating the group semantic mean is as follows: ; in, Let be the group semantic mean at time t; The total number of users interacting at time t; Let be the semantic vector of the i-th user at time t.

[0039] Subsequently, the individual deviation of each user is quantified by calculating the difference between each user's semantic vector and the group's semantic mean.

[0040] The specific method for calculating individual deviation is as follows: ; in, Let be the individual deviation of the i-th user at time t; This is the symbol for the 2-norm.

[0041] Individual deviations are compared with preset deviation thresholds to determine whether a user's semantic behavior is abnormal. The deviation threshold is a pre-set value used to distinguish whether a user's individual deviation level warrants further attention. This threshold can be dynamically adjusted based on historical data analysis, expert experience, or machine learning models to adapt to different scenarios and risk levels.

[0042] When a user's individual deviation exceeds a preset deviation threshold, it indicates that the user's semantic behavior may be abnormal and requires further analysis; therefore, the user is classified as a user requiring detailed investigation.

[0043] Conversely, when a user's individual deviation does not exceed the deviation threshold, it indicates that the user's semantic behavior is within the normal range and the risk is low, so the user is classified as a coarse-screened user.

[0044] Ultimately, different monitoring strategies will be adopted for users identified as subject to detailed scrutiny and those identified as subject to broad screening. For example, for users subject to detailed scrutiny, a stricter and more frequent monitoring mechanism can be initiated to conduct more in-depth behavioral analysis or real-time alerts. This could involve calculating the manifold hedging score and comprehensive security score for users subject to detailed scrutiny to deeply analyze the user interaction content information of the corresponding AI system. Conversely, for users identified as subject to broad screening, a more lenient monitoring strategy can be adopted. For example, only the manifold hedging score for users subject to detailed scrutiny can be calculated to monitor the AI ​​system based on the manifold hedging score, thereby saving computing resources and improving overall monitoring efficiency. Through this differentiated monitoring strategy, more accurate and efficient risk management of AI systems can be achieved.

[0045] In step S102, the semantic vector is input into the compliance constraint topology graph model for node filtering. By calculating the similarity between the semantic vector and the semantic vector of each node in the graph (compliance constraint topology graph model), the nodes that are most semantically related to the user interaction content information are identified, thereby obtaining the compliance manifold nodes and violation manifold nodes corresponding to the semantic vector. These manifold nodes (compliance manifold nodes and violation manifold nodes) represent the potential correlation points of the user interaction content information in the compliance and violation semantic space.

[0046] Based on the compliant semantic vectors of compliant manifold nodes and the non-compliant semantic vectors of non-compliant manifold nodes, and combined with preset semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients, similarity is calculated for semantic vectors. The semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients serve as weighting factors to adjust the contribution of different semantic nodes to the density field intensity. Through this calculation, the compliant and non-compliant topological density field intensities corresponding to the semantic vectors can be obtained, thereby quantifying the tendency of user interaction content information in the compliant and non-compliant dimensions. For example, the semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients can be obtained by calculating the cosine similarity between the semantic vector and the plotted vector, the reciprocal of the Euclidean distance, or through probability density functions (such as the Gaussian kernel function), respectively, to quantify the "density" or "degree of belonging" of the current semantic vector in the compliant or non-compliant semantic space.

[0047] The strengths of both lawful and illegal topological density fields can be represented using the Gibbs-Boltzmann potential. The specific formula for calculating the strength of the lawful topological density field is as follows: ; in, The compliant topological density field strength; k is the nearest neighbor anchor coefficient, typically taken as 30; z is the semantic vector; For compliant semantic sets; This means selecting the top k compliance semantics that are most similar to the semantic vector from the compliance semantic set, i.e., the compliance semantics corresponding to the compliance manifold nodes; Let be the compliance vector corresponding to the nth compliant manifold node; This is the semantic compliance density distribution coefficient, with a value range of [value range missing]. ; It is a natural exponential function; This represents the cosine similarity between the semantic vector and the compliance vector corresponding to the nth compliance manifold node. This represents the similarity offset between the semantic vector and the compliance vector corresponding to the nth compliant manifold node. Wherein, the semantic compliance density distribution coefficient... (and semantic violation density distribution coefficient) ) is determined based on the physical meaning of the Gibbs-Boltzmann distribution: when (or The smaller the value of ) (i.e., approaching 0), the more the distribution of semantic vectors and nodes of the compliant manifold tends to the Dirac function (a function that is zero everywhere except the origin and has a global integral of 1), meaning the density distribution is more concentrated (more sensitive to nearest neighbors). At this point, the system degenerates into a hard threshold decision. (or As the value increases (i.e., approaches 1), the distribution of semantic vectors and compliant manifold nodes becomes more uniform, resulting in a smoother density distribution. At this point, the system loses its discriminative ability. Therefore, when... (and The preferred value for ) is 0.05.

[0048] The specific formula for calculating the intensity of the illegal topological density field is as follows: ; in, For the intensity of the illegal topological density field; A collection of non-compliant semantics; This means selecting the k most similar non-violation semantics to the semantic vector from the set of non-violation semantics, that is, the non-violation semantics corresponding to the nodes of the non-violation manifold; Let be the violation vector corresponding to the m-th violation manifold node; This is the semantic violation density distribution coefficient, with a value range of [value range missing]. .

[0049] Specifically, the K-Means clustering algorithm was used to extract cluster centroids from both the compliant and non-compliant semantic sets as simplified anchor points, significantly reducing storage and computational overhead while maintaining detection accuracy. Experiments show that, on the benchmark for jailbreak detection, using 30 K-Means centroid anchor points (i.e., k=30) achieves an AUC (Area Under the Curve) of 0.9909, surpassing the 0.9826 of all anchor points. Specific experimental data are shown in Table 1 below. Table 1. Comparison of benchmark performance under different numbers of centroid anchor points

[0050] As shown in Table 1, when k=30, the jailbreak detection score is 0.9909, indicating that when using 30 centroid anchor points, K-Means clustering can effectively remove redundant information, enhance the ability to focus on compliance vectors and violation vectors, and reduce storage overhead and improve memory access efficiency while maintaining an AUC of 0.9909.

[0051] In step S103, based on the topological density field strength information and combined with preset compliance / violation balance parameter information, a manifold hedging score is calculated to obtain the manifold hedging score corresponding to the user interaction content information. The compliance / violation balance parameter information includes a balance factor, a scaling factor, and an offset factor, which are parameters used to finely adjust the manifold hedging score. The balance factor is used to adjust the relative weights of the violation and compliance topological density field strengths in the manifold hedging score to adapt to the differences in the degree of attention paid to violation and compliance in different application scenarios; the scaling factor is used to control the sharpness of the manifold hedging score transitioning from "compliant" to "violation," ensuring the comparability and consistency of the scores; and the offset factor is used to shift the manifold hedging score overall to set a baseline or adjust the system's sensitivity.

[0052] The specific formula for calculating the manifold hedging score is as follows: ; in, Scoring for manifold hedging; Use the Sigmoid activation function; As a balancing factor, its value range is generally [value range missing]. ; This is the scaling factor, and its value range is generally [value range missing]. ; This is the offset factor, and its value range is generally [value range missing]. Among them, the balance factor Scaling factor and offset factor It can be configured according to actual needs. Therefore, the higher the calculated intensity of the illegal topological density field, the easier it is to intercept.

[0053] In an optional embodiment, specific calculation formulas for the compliant topological density field strength, the non-compliant topological density field strength, and the manifold hedging score can be set in the compliant constraint topological graph model, so that the compliant constraint topological graph model can calculate the semantic compliance score, the semantic non-compliant score, and the manifold hedging score while identifying compliant manifold nodes and non-compliant manifold nodes.

[0054] Specifically, the temporal evolution feature information includes a semantic long-term monitoring score, a setting offset detection score, and a malicious premeditation detection score. Optionally, in step S104, based on the manifold hedging score, a comprehensive scoring process is performed in conjunction with the temporal evolution feature information corresponding to the user interaction content information to obtain a comprehensive security score corresponding to the user interaction content information, including: Historical monitoring and scoring are performed on the semantic vectors corresponding to user interaction content information to obtain long-term semantic monitoring scores and set offset detection scores for user interaction content information. Malicious premeditation detection processing is performed on the semantic vectors corresponding to user interaction content information to obtain a malicious premeditation detection score for user interaction content information. Based on manifold hedging score, long-term semantic monitoring score, setting offset detection score, and malicious premeditation detection score, a weighted summation is performed to obtain a comprehensive security score for user interaction content information.

[0055] Specifically, in step S104, the semantic vector corresponding to the user interaction content information is subjected to historical monitoring and scoring processing to obtain the long-term semantic monitoring score and the set offset detection score of the user interaction content information, including: Based on the preset short-term decay coefficient and the preset long-term decay coefficient, the short-term memory state and long-term memory state are calculated for the semantic vector corresponding to the user interaction content information to obtain the short-term memory state parameters and long-term memory state parameters of the user interaction content information. The hazard index is calculated based on the short-term memory state parameters and the Laplace matrix of the compliance constraint topology graph model. Based on long-term memory state parameters, the hazard index is calculated by combining the Laplace matrix of the compliance constraint topology graph model, and the long-term hazard index is obtained. The maximum value among the short-term memory state parameters and the long-term memory state parameters is determined as the semantic long-term monitoring score of user interaction content information; Based on the short-term memory state parameters and the preset baseline state vector, the difference is calculated to obtain the setting offset detection score of the user interaction content information.

[0056] It's important to note that the short-term decay coefficient measures the impact of recent user interactions on the current state of the AI ​​system, capturing immediate changes in its behavior. The long-term decay coefficient, on the other hand, assesses the cumulative impact of historical user interactions on the long-term behavioral patterns of the AI ​​system, reflecting its stability and trends. Generally, the short-term decay coefficient is smaller than the long-term decay coefficient. The short-term decay coefficient is typically set to a smaller value to ensure that older interactions still have a sustained but gradually diminishing impact on short-term memory parameters, such as a range of 0.3 to 0.7. The long-term decay coefficient, however, is set to a larger value to make the impact of recent interactions more significant on long-term memory parameters, such as a range of 0.999 to 0.9999.

[0057] The Laplacian matrix is ​​a commonly used matrix in graph theory to analyze the structural relationships between semantic vectors. It can identify anomalous patterns or structural shifts in the semantic space and can be calculated using the compliance constraint topological graph model described above. The Laplacian matrix is ​​existing technology and will not be discussed in detail here.

[0058] In step S104, a dual-time-scale exponential moving average (EMA) calculation method can be used to simultaneously capture the short-term fluctuations and long-term trends of the data (semantic vector), thereby calculating the long-term semantic monitoring score and the set offset detection score. First, based on the semantic vector corresponding to the user interaction content information, the short-term decay coefficient and long-term decay coefficient are combined to calculate the short-term memory state parameters and long-term memory state parameters of the artificial intelligence system, thereby determining the behavioral characteristics of the artificial intelligence system within a shorter time window and the cumulative behavioral characteristics over a longer time span.

[0059] The specific formula for calculating the short-term memory state parameters is as follows: ; in, Let be the short-term memory state parameters at time t; This is the short-term decay coefficient; These are the short-term memory state parameters at time t-1; when t=1, .

[0060] The specific formula for calculating long-term memory state parameters is as follows: ; in, Let be the long-term memory state parameters at time t; This is the long-term attenuation coefficient; These are the long-term memory state parameters at time t-1; when t=1, .

[0061] Then, based on the short-term memory state parameters and the Laplace matrix, a short-term hazard index can be calculated. This short-term hazard index can assess whether there are potential, immediate risks in the current or recent behavior of the artificial intelligence system.

[0062] Similarly, based on long-term memory state parameters and combined with the Laplace matrix, a long-term risk index can be obtained, which can assess whether there are persistent and cumulative risks in the long-term operation of artificial intelligence systems.

[0063] The specific formula for calculating the short-term risk index is as follows: ; in, This is a short-term risk index; It is a Laplace matrix.

[0064] The formula for calculating the long-term risk index is as follows: ; in, This is a long-term risk index.

[0065] The maximum value between the short-term and long-term memory state parameters is determined as the long-term semantic monitoring score of the artificial intelligence system. This aims to ensure that the long-term semantic monitoring score can comprehensively reflect the system's highest risk level in both the short-term and long-term time dimensions. Thus, by combining the dual-timescale EMA with the Graph Laplace energy (Graph Laplace matrix), real-time detection of multi-turn semantic drift is achieved, improving the internal performance of the computer system when processing multi-turn dialogues.

[0066] The specific formula for calculating the offset detection score is as follows: ; in, To set an offset detection score; This is the baseline state vector.

[0067] Thus, by combining dual-timescale EMA with Thulaplace energy, real-time detection of multi-turn semantic drift is achieved, improving the internal performance of the computer system when processing multi-turn dialogues.

[0068] Specifically, in step S104, the semantic vector corresponding to the user interaction content information is subjected to malicious premeditation detection processing to obtain a malicious premeditation detection score for the user interaction content information, including: Based on the preset sensitive topic classification rules, sensitive topic features are extracted from the semantic vector corresponding to the user interaction content information to obtain the sensitive topic feature vector of the artificial intelligence system, and the number of times the sensitive topic appears in the sensitive topic feature vector is determined. Based on the preset user rejection intent recognition rules, rejection features are extracted from the semantic vector corresponding to the user interaction content information to obtain the user's rejection semantic feature vector, and the number of times the rejection semantic appears in the rejection semantic feature vector is determined. Based on the rejection semantic feature vector, sensitive topic feature vector, the number of times sensitive topics appear and the number of times rejection semantics appear, we conduct active sensitive topic statistics and repeated mention statistics to obtain the number of times the AI ​​system actively mentions sensitive topics, and the number of times the AI ​​system mentions any sensitive topic again after the user rejects any sensitive topic. A malicious intent detection score for user interaction content information is obtained by calculating the ratio of the number of times sensitive topics are actively mentioned to the number of times sensitive topics are repeatedly mentioned.

[0069] It should be noted that the preset sensitive topic classification rules are a set of rules used to identify whether sensitive or illegal topics exist in user interaction content. These rules can be implemented based on technologies such as keyword matching, topic models, and deep learning classifiers, and can accurately identify semantic features related to sensitive topics from the semantic vectors converted from user interaction content information. By extracting sensitive topic features from the semantic vectors, a sensitive topic feature vector representing the sensitive topic can be obtained, and the number of times the sensitive topic feature vector appears in the user interaction history can be recorded. For example, when the artificial intelligence system mentions content involving violence, pornography, or fraud, the corresponding sensitive topic feature vector will be extracted and counted.

[0070] Pre-defined user refusal intent recognition rules refer to a set of rules used to determine whether a user exhibits refusal, avoidance, denial, or concealment of their true intent. These rules can be implemented based on techniques such as sentiment analysis, syntactic structure analysis, specific phrase recognition, or contextual understanding, and can identify refusal semantic features from semantic vectors that indicate a user's attempt to hide or deny their behavioral intent. By extracting refusal features from the semantic vectors, a refusal semantic feature vector representing the user's refusal intent and the number of occurrences of the refusal semantics within that feature vector can be obtained. For example, when a user replies with statements such as "I was just asking," "It's none of your business," or "This is none of your concern" after being asked a sensitive topic, the refusal semantic feature vector can be identified and counted.

[0071] In step S104, malicious premeditated behavior is often not a one-off, isolated event, but rather unfolds gradually through a series of probing, concealment, and evasion behaviors. By extracting feature vectors of sensitive topics and semantic rejection, and analyzing their temporal relationship and density, potential malicious intent in the behavioral patterns of artificial intelligence systems can be captured. For example, when an AI system repeatedly mentions sensitive topics and exhibits rejection or evasion each time it is guided back to the right track by the user, this pattern strongly suggests premeditation. The cumulative number of times sensitive topics and semantic rejection appear further strengthens the judgment of this premeditation, enabling the distinction between accidental verbal slips and purposeful malicious behavior. Therefore, by statistically analyzing the sensitive topic feature vector of the AI ​​system and the number of times sensitive topics appear in the sensitive topic feature vector, and by statistically analyzing the user's rejection semantic feature vector and the number of times rejection semantics appear in the rejection semantic feature vector, and based on the rejection semantic feature vector, sensitive topic feature vector, number of times sensitive topics appear, and number of times rejection semantics appear, we can conduct proactive sensitive topic statistics and repeated mention statistics to obtain the number of times the AI ​​system proactively mentions sensitive topics, and the number of times the AI ​​system mentions any sensitive topic again after the user rejects any sensitive topic. Thus, based on the ratio of the number of proactive mentions of sensitive topics to the number of repeated mentions of sensitive topics, we can calculate the malicious premeditation detection score of the AI ​​system, thereby identifying indicators of whether the AI ​​system has premeditated or planned malicious behavior.

[0072] For example, the specific formula for calculating the malicious premeditation detection score is as follows: ; Among them, C represents the score for detecting malicious premeditation; The number of times an artificial intelligence system proactively guides the discussion of sensitive topics; This refers to the number of times an AI system attempts to steer the conversation toward sensitive topics after being rejected by the user.

[0073] Specifically, in step S104, the semantic long-term monitoring score can capture the cumulative effect and long-term trend of user behavior, avoiding the neglect of potential long-term risks due to short-term compliance; the offset detection score can promptly detect abnormal changes in user behavior patterns, which is of great significance for identifying sudden violations or risky behaviors; and the malicious intent detection score focuses on identifying users' deep-seated malicious intentions, which is crucial for preventing organized and planned attacks or abuses. By weighting and integrating these multi-dimensional temporal features with the manifold hedging score, a comprehensive and integrated risk assessment indicator, namely the comprehensive security score, is formed. This comprehensive security score can more comprehensively and dynamically reflect the security status of the artificial intelligence system, thereby improving the depth and breadth of monitoring. In addition, different weights can be assigned to each score according to its importance in the overall security assessment, and the weighted summation can be performed to flexibly adjust the contribution of each factor to the final comprehensive security score.

[0074] Specifically, in step S105, based on the comprehensive safety score, the artificial intelligence system is monitored to obtain the monitoring results of the artificial intelligence system, including: The system determines whether the comprehensive security score is greater than or equal to the preset security control threshold, the manifold hedging score is greater than or equal to the preset semantic violation and compliance threshold, the semantic long-term monitoring score is greater than or equal to the preset long-term monitoring threshold, the set offset detection score is greater than or equal to the preset set offset threshold, and the malicious premeditation detection score is greater than or equal to the preset malicious premeditation threshold. When the comprehensive security score is greater than or equal to the security control threshold, or the manifold hedging score is greater than or equal to the semantic violation compliance threshold, or the semantic long-term monitoring score is greater than or equal to the long-term monitoring threshold, or the set offset detection score is greater than or equal to the set offset threshold, or the malicious premeditation detection score is greater than or equal to the malicious premeditation threshold, the artificial intelligence system is intervened and controlled according to the preset intervention strategy to obtain the security monitoring results of the artificial intelligence system. When the overall security score is less than the security control threshold, the manifold hedging score is less than the semantic violation compliance threshold, the semantic long-term monitoring score is less than the long-term monitoring threshold, the set offset detection score is less than the set offset threshold, and the malicious premeditation detection score is less than the malicious premeditation threshold, it is determined that no intervention or control is required for the artificial intelligence system, and the security monitoring result of the artificial intelligence system is obtained.

[0075] In step S105, to achieve more refined monitoring, in addition to determining whether the overall security score is greater than or equal to the preset security control threshold, each independent score is also determined separately: whether the manifold hedging score is greater than or equal to the preset semantic violation compliance threshold, whether the semantic long-term monitoring score is greater than or equal to the preset long-term monitoring threshold, whether the set offset detection score is greater than or equal to the preset set offset threshold, and whether the malicious premeditation detection score is greater than or equal to the preset malicious premeditation threshold. These preset thresholds (security control threshold, semantic violation compliance threshold, long-term monitoring threshold, set offset threshold, and malicious premeditation threshold) are determined based on actual application scenarios and security requirements, and are used to define the critical points for different security risk levels.

[0076] Therefore, when the overall security score is greater than or equal to the security control threshold, it indicates that the overall security risk of the artificial intelligence system is high, requiring intervention. Furthermore, even if the overall score does not reach the threshold, if any individual score (e.g., manifold hedging score, long-term semantic monitoring score, setpoint offset detection score, or malicious premeditation detection score) is greater than or equal to its corresponding preset threshold, it also indicates a significant risk in that specific security dimension, requiring intervention. Intervention strategies can be preset according to the type and severity of the risk. For example, these could include restricting the AI ​​system's interaction permissions with users, reviewing suspicious content, issuing alerts to administrators, suspending some system functions, or initiating deeper audit processes. By implementing these intervention controls, the security risks of the AI ​​system can be effectively reduced, and corresponding security monitoring results can be obtained.

[0077] Conversely, when the overall security score is less than the security control threshold, and all individual scores (i.e., manifold hedging score, semantic long-term monitoring score, setting offset detection score, and malicious premeditation detection score) are less than their corresponding preset thresholds, the artificial intelligence system is considered to be in a safe state and no intervention or control is required, thus obtaining a security monitoring result that does not require intervention.

[0078] As shown above, the monitoring method for this artificial intelligence system involves acquiring user interaction content information from the AI ​​system, calculating the topological density field strength based on this information and a pre-defined compliance constraint topology model, obtaining the topological density field strength information corresponding to the user interaction content information, calculating the manifold hedging score based on the topological density field strength information and pre-defined compliance / violation balance parameters, obtaining the manifold hedging score corresponding to the user interaction content information, performing comprehensive scoring based on the manifold hedging score and the time-series evolution characteristics corresponding to the user interaction content information, obtaining the comprehensive security score corresponding to the user interaction content information, and monitoring the AI ​​system based on the comprehensive security score. The monitoring results of the artificial intelligence system are obtained through reasoning. Then, by using a pre-set compliance constraint topology graph model and pre-set compliance / violation balance parameter information to calculate the manifold hedging score, combined with the temporal evolution characteristics of user interaction content information, a comprehensive scoring process is performed to obtain a comprehensive security score. This comprehensive security score is then used to monitor the artificial intelligence system, addressing the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise-level applications for artificial intelligence systems. Through a multi-dimensional scoring mechanism, deep semantic understanding and risk assessment of user interaction content are achieved, improving the monitoring accuracy of the artificial intelligence system.

[0079] refer to Figure 2 This application provides a monitoring device for an artificial intelligence system, used for monitoring the artificial intelligence system, including: Module 1 is used to acquire user interaction content information of the artificial intelligence system; The first calculation module 2 is used to calculate the topological density field strength based on the user interaction content information and the corresponding preset compliance constraint topological graph model of the artificial intelligence system, so as to obtain the topological density field strength information corresponding to the user interaction content information. The second calculation module 3 is used to calculate the manifold hedging score based on the topological density field strength information and the preset compliance and non-compliance balance parameter information, so as to obtain the manifold hedging score corresponding to the user interaction content information. The comprehensive scoring module 4 is used to perform comprehensive scoring based on manifold hedging scoring and combined with the temporal evolution feature information corresponding to user interaction content information to obtain a comprehensive security score corresponding to user interaction content information. Monitoring module 5 is used to monitor and process the artificial intelligence system based on the comprehensive security score, and obtain the monitoring results of the artificial intelligence system.

[0080] The monitoring device of this artificial intelligence system, after acquiring user interaction content information, calculates a manifold hedging score using a preset compliance constraint topology graph model and preset compliance / violation balance parameters. This score, combined with the temporal evolution characteristics of the user interaction content, is then used to perform comprehensive scoring to obtain a comprehensive security score. This comprehensive security score is then used to monitor the artificial intelligence system. This addresses the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise applications. Through a multi-dimensional scoring mechanism, it achieves deep semantic understanding and risk assessment of user interaction content, thereby improving the monitoring accuracy of the artificial intelligence system.

[0081] Specifically, when module 1 is executed, it acquires the dialogue text between the user and AI systems such as chatbots and intelligent customer service, thus obtaining user interaction content information from the AI ​​systems. This user interaction content information refers to all data generated during the interaction between the user and the AI ​​system, including text, voice, and images. This data carries the intent and behavioral information of both the user and the AI ​​system.

[0082] Specifically, the topology density field strength includes compliant topology density field strength and non-compliant topology density field strength. Optionally, when the first calculation module 2 calculates the topology density field strength based on the user interaction content information and the corresponding preset compliant constraint topology graph model of the artificial intelligence system to obtain the topology density field strength information corresponding to the user interaction content information, it executes: Obtain the pre-defined compliance constraint topology graph model corresponding to the artificial intelligence system; the compliance constraint topology graph model is constructed in advance by taking each compliance semantic in the pre-defined compliance semantic set and each violation semantic in the pre-defined violation semantic set as nodes, taking the pre-defined compliance association relationship between each compliance semantic and the pre-defined violation association relationship between each violation semantic as edges, configuring the compliance semantic vector corresponding to the compliance semantic and the violation semantic vector corresponding to the violation semantic for each node, and assigning values ​​to each edge by the pre-defined compliance quantification assignment corresponding to the compliance association relationship and the pre-defined violation quantification assignment corresponding to the violation association relationship; Convert user interaction content information into semantic vectors; The semantic vectors are input into the compliance constraint topology graph model for node filtering, resulting in the compliance manifold nodes and violation manifold nodes corresponding to the semantic vectors. Based on the compliant semantic vector of the compliant manifold node and the non-compliant semantic vector of the non-compliant manifold node, and combined with the preset semantic compliance density distribution coefficient and semantic non-compliant density distribution coefficient, the similarity of the semantic vectors is calculated to obtain the compliant topological density field intensity and the non-compliant topological density field intensity corresponding to the semantic vector.

[0083] It should be noted that the topology density field strength is subdivided into compliant topology density field strength and non-compliant topology density field strength, aiming to quantitatively evaluate user interaction content information from two dimensions: compliance and non-compliance. Specifically, the compliant topology density field strength measures the degree to which user interaction content information conforms to compliant semantics, while the non-compliant topology density field strength measures its degree to which it conforms to non-compliant semantics.

[0084] When the first calculation module 2 is executed, it acquires a pre-constructed knowledge graph, namely a compliance constraint topology graph model. The construction process includes: using each compliance semantic in a pre-defined set of compliance semantics and each violation semantic in a pre-defined set of violation semantics as nodes of the graph; using pre-defined compliance relationships and pre-defined violation relationships between compliance semantics as edges of the graph; configuring compliance semantic vectors and violation semantic vectors for each node; and assigning values ​​to each edge based on pre-defined compliance quantifications and violation quantifications corresponding to compliance and violation relationships. Thus, this topology graph model (i.e., the compliance constraint topology graph model) can comprehensively represent compliance and violation semantics and their interrelationships, providing a structured knowledge foundation for subsequent density field intensity calculations. The compliance semantic set refers to a pre-determined set of semantics representing compliance, using rules that identify legal, positive, and constructive content, such as "normal consultation" and "product introduction." The violation semantic set refers to a pre-determined set of semantics representing violation, using rules that identify illegal, negative, and harmful content, such as "privacy leak" and "sensitive political speech." Furthermore, the pre-defined compliance relationships and their corresponding compliance quantifications, as well as the pre-defined violation relationships and their corresponding violation quantifications, can be obtained based on the association criteria and scoring criteria set in patent opinions and related books.

[0085] In practical implementation, existing technologies such as natural language processing can be used. For example, pre-trained embedding models (such as Word2Vec and GloVe) or pre-defined contextual embedding models (such as BERT and GPT series models) can be used to encode the text content, mapping each word or sentence to a point in a high-dimensional vector space. This converts the user interaction content information into a numerical vector that can represent its semantic features, i.e., a semantic vector. This vectorized representation enables computers to understand and process the deeper meaning of the text.

[0086] For example, the semantic vector is calculated as follows: ; in, Let be the semantic vector corresponding to the user interaction content information at time t; This refers to user interaction information at time t. This is an embedding model (one of the ways to calculate semantic vectors, which can be adjusted according to actual needs).

[0087] Optionally, after converting the user interaction content information into a semantic vector, the first calculation module 2 also performs: The group semantic mean of all users is calculated based on the semantic vectors of all users. The difference between the semantic vector of each user and the group semantic mean is calculated to obtain the individual deviation of each user. Compare the individual deviation with a preset deviation threshold; When an individual's deviation exceeds a preset deviation threshold, the user corresponding to that individual's deviation will be identified as a user subject to detailed investigation. When an individual's deviation is less than or equal to a preset deviation threshold, the user corresponding to that individual's deviation will be identified as a coarse-screened user. Different monitoring strategies are adopted for users who perform detailed checks and users who perform coarse checks, and the artificial intelligence system is used to monitor these users.

[0088] It should be noted that the group semantic mean is the average or median of the semantic vectors of all user interactions within a specific time period or across the entire user group. This mean represents the overall semantic behavior pattern or "normal" behavior baseline of the user group, and can be calculated by aggregating the semantic vectors of all users and taking the average, median, or other statistical methods. Individual deviation refers to the distance or degree of difference between a single user's semantic vector and the group semantic mean. This deviation can be calculated using various distance metrics, such as Euclidean distance, cosine similarity (inverted or converted to distance), Mahalanobis distance, etc., to quantify the degree of deviation of user behavior from the group's norm.

[0089] Specifically, when the first calculation module 2 is executed, after obtaining the semantic vectors, it calculates the average semantic value of all users by averaging the semantic vectors of all users. This average semantic value represents the average semantic characteristics of all current user interaction behaviors and can be regarded as a baseline of "normal" or "typical" user behavior.

[0090] The specific method for calculating the group semantic mean is as follows: ; in, Let be the group semantic mean at time t; The total number of users interacting at time t; Let be the semantic vector of the i-th user at time t.

[0091] Subsequently, the individual deviation of each user is quantified by calculating the difference between each user's semantic vector and the group's semantic mean.

[0092] The specific method for calculating individual deviation is as follows: ; in, Let be the individual deviation of the i-th user at time t; This is the symbol for the 2-norm.

[0093] Individual deviations are compared with preset deviation thresholds to determine whether a user's semantic behavior is abnormal. The deviation threshold is a pre-set value used to distinguish whether a user's individual deviation level warrants further attention. This threshold can be dynamically adjusted based on historical data analysis, expert experience, or machine learning models to adapt to different scenarios and risk levels.

[0094] When a user's individual deviation exceeds a preset deviation threshold, it indicates that the user's semantic behavior may be abnormal and requires further analysis; therefore, the user is classified as a user requiring detailed investigation.

[0095] Conversely, when a user's individual deviation does not exceed the deviation threshold, it indicates that the user's semantic behavior is within the normal range and the risk is low, so the user is classified as a coarse-screened user.

[0096] Ultimately, different monitoring strategies will be adopted for users identified as subject to detailed scrutiny and those identified as subject to broad screening. For example, for users subject to detailed scrutiny, a stricter and more frequent monitoring mechanism can be initiated to conduct more in-depth behavioral analysis or real-time alerts. This could involve calculating the manifold hedging score and comprehensive security score for users subject to detailed scrutiny to deeply analyze the user interaction content information of the corresponding AI system. Conversely, for users identified as subject to broad screening, a more lenient monitoring strategy can be adopted. For example, only the manifold hedging score for users subject to detailed scrutiny can be calculated to monitor the AI ​​system based on the manifold hedging score, thereby saving computing resources and improving overall monitoring efficiency. Through this differentiated monitoring strategy, more accurate and efficient risk management of AI systems can be achieved.

[0097] When the first calculation module 2 is executed, it inputs the semantic vector into the compliance constraint topology graph model for node filtering. By calculating the similarity between the semantic vector and the semantic vector of each node in the graph (compliance constraint topology graph model), it identifies the nodes that are most semantically relevant to the user interaction content information, thereby obtaining the compliance manifold nodes and violation manifold nodes corresponding to the semantic vector. These manifold nodes (compliance manifold nodes and violation manifold nodes) represent the potential correlation points of user interaction content information in the compliance and violation semantic space.

[0098] Based on the compliant semantic vectors of compliant manifold nodes and the non-compliant semantic vectors of non-compliant manifold nodes, and combined with preset semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients, similarity is calculated for semantic vectors. The semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients serve as weighting factors to adjust the contribution of different semantic nodes to the density field intensity. Through this calculation, the compliant and non-compliant topological density field intensities corresponding to the semantic vectors can be obtained, thereby quantifying the tendency of user interaction content information in the compliant and non-compliant dimensions. For example, the semantic compliance density distribution coefficients and semantic non-compliance density distribution coefficients can be obtained by calculating the cosine similarity between the semantic vector and the plotted vector, the reciprocal of the Euclidean distance, or through probability density functions (such as the Gaussian kernel function), respectively, to quantify the "density" or "degree of belonging" of the current semantic vector in the compliant or non-compliant semantic space.

[0099] The strengths of both lawful and illegal topological density fields can be represented using the Gibbs-Boltzmann potential. The specific formula for calculating the strength of the lawful topological density field is as follows: ; in, The compliant topological density field strength; k is the nearest neighbor anchor coefficient, typically taken as 30; z is the semantic vector; For compliant semantic sets; This means selecting the top k compliance semantics that are most similar to the semantic vector from the compliance semantic set, i.e., the compliance semantics corresponding to the compliance manifold nodes; Let be the compliance vector corresponding to the nth compliant manifold node; This is the semantic compliance density distribution coefficient, with a value range of [value range missing]. ; It is a natural exponential function; This represents the cosine similarity between the semantic vector and the compliance vector corresponding to the nth compliance manifold node. This represents the similarity offset between the semantic vector and the compliance vector corresponding to the nth compliance manifold node.

[0100] The specific formula for calculating the intensity of the illegal topological density field is as follows: ; in, For the intensity of the illegal topological density field; A collection of non-compliant semantics; This means selecting the k most similar non-violation semantics to the semantic vector from the set of non-violation semantics, that is, the non-violation semantics corresponding to the nodes of the non-violation manifold; Let be the violation vector corresponding to the m-th violation manifold node; This is the semantic violation density distribution coefficient, with a value range of [value range missing]. .

[0101] Specifically, the K-Means clustering algorithm was used to extract cluster centroids from both the compliant and non-compliant semantic sets as simplified anchor points, significantly reducing storage and computational overhead while maintaining detection accuracy. Experiments show that, on the benchmark for jailbreak detection, using 30 K-Means centroid anchor points (i.e., k=30) achieves an AUC of 0.9909, surpassing the 0.9826 of all anchor points. Specific experimental data are shown in Table 1 below.

[0102] As shown in Table 1, when k=30, the jailbreak detection score is 0.9909, indicating that when using 30 centroid anchor points, K-Means clustering can effectively remove redundant information and enhance the ability to focus on compliance vectors and violation vectors.

[0103] During execution, the second calculation module 3 calculates the manifold hedging score based on the topological density field strength information and the preset compliance / violation balance parameters, obtaining the manifold hedging score corresponding to the user interaction content information. The compliance / violation balance parameters include a balance factor, a scaling factor, and an offset factor, which are used to fine-tune the manifold hedging score. The balance factor adjusts the relative weights of the violation and compliance topological density field strengths in the manifold hedging score to adapt to the different levels of focus on compliance and non-compliance in different application scenarios; the scaling factor controls the sharpness of the manifold hedging score transitioning from "compliant" to "violation," ensuring the comparability and consistency of the scores; and the offset factor is used to shift the manifold hedging score overall to set a baseline or adjust the system's sensitivity.

[0104] The specific formula for calculating the manifold hedging score is as follows: ; in, Scoring for manifold hedging; Use the Sigmoid activation function; As a balancing factor, its value range is generally [value range missing]. ; This is the scaling factor, and its value range is generally [value range missing]. ; This is the offset factor, and its value range is generally [value range missing]. Among them, the balance factor Scaling factor and offset factor It can be configured according to actual needs. Therefore, the higher the calculated intensity of the illegal topological density field, the easier it is to intercept.

[0105] In an optional embodiment, specific calculation formulas for the compliant topological density field strength, the non-compliant topological density field strength, and the manifold hedging score can be set in the compliant constraint topological graph model, so that the compliant constraint topological graph model can calculate the semantic compliance score, the semantic non-compliant score, and the manifold hedging score while identifying compliant manifold nodes and non-compliant manifold nodes.

[0106] Specifically, the temporal evolution feature information includes a semantic long-term monitoring score, a setting offset detection score, and a malicious premeditation detection score. Optionally, when the comprehensive scoring module 4 performs comprehensive scoring processing based on the manifold hedging score and combines it with the temporal evolution feature information corresponding to the user interaction content information to obtain a comprehensive security score corresponding to the user interaction content information, it executes the following: Historical monitoring and scoring are performed on the semantic vectors corresponding to user interaction content information to obtain long-term semantic monitoring scores and set offset detection scores for user interaction content information. Malicious premeditation detection processing is performed on the semantic vectors corresponding to user interaction content information to obtain a malicious premeditation detection score for user interaction content information. Based on manifold hedging score, long-term semantic monitoring score, setting offset detection score, and malicious premeditation detection score, a weighted summation is performed to obtain a comprehensive security score for user interaction content information.

[0107] Specifically, when the comprehensive scoring module 4 performs historical monitoring and scoring on the semantic vectors corresponding to user interaction content information to obtain the long-term semantic monitoring score and the set offset detection score for user interaction content information, it executes the following: Based on the preset short-term decay coefficient and the preset long-term decay coefficient, the short-term memory state and long-term memory state are calculated for the semantic vector corresponding to the user interaction content information to obtain the short-term memory state parameters and long-term memory state parameters of the user interaction content information. The hazard index is calculated based on the short-term memory state parameters and the Laplace matrix of the compliance constraint topology graph model. Based on long-term memory state parameters, the hazard index is calculated by combining the Laplace matrix of the compliance constraint topology graph model, and the long-term hazard index is obtained. The maximum value among the short-term memory state parameters and the long-term memory state parameters is determined as the semantic long-term monitoring score of user interaction content information; Based on the short-term memory state parameters and the preset baseline state vector, the difference is calculated to obtain the setting offset detection score of the user interaction content information.

[0108] It's important to note that the short-term decay coefficient measures the impact of recent user interactions on the current state of the AI ​​system, capturing immediate changes in its behavior. The long-term decay coefficient, on the other hand, assesses the cumulative impact of historical user interactions on the long-term behavioral patterns of the AI ​​system, reflecting its stability and trends. Generally, the short-term decay coefficient is smaller than the long-term decay coefficient. The short-term decay coefficient is typically set to a smaller value to ensure that older interactions still have a sustained but gradually diminishing impact on short-term memory parameters, such as a range of 0.3 to 0.7. The long-term decay coefficient, however, is set to a larger value to make the impact of recent interactions more significant on long-term memory parameters, such as a range of 0.999 to 0.9999.

[0109] The Laplacian matrix is ​​a commonly used matrix in graph theory to analyze the structural relationships between semantic vectors. It can identify anomalous patterns or structural shifts in the semantic space and can be calculated using the compliance constraint topological graph model described above. The Laplacian matrix is ​​existing technology and will not be discussed in detail here.

[0110] When executing the comprehensive scoring module 4, an Exponential Moving Average (EMA) calculation method with dual time scales can be used to simultaneously capture the short-term fluctuations and long-term trends of the data (semantic vectors), thereby calculating the long-term semantic monitoring score and the set offset detection score. First, based on the semantic vectors corresponding to the user interaction content information, short-term decay coefficients and long-term decay coefficients are combined to calculate the short-term memory state parameters and long-term memory state parameters of the artificial intelligence system. This determines the behavioral characteristics of the artificial intelligence system within a short time window and its cumulative behavioral characteristics over a longer time span.

[0111] The specific formula for calculating the short-term memory state parameters is as follows: ; in, Let be the short-term memory state parameters at time t; This is the short-term decay coefficient; These are the short-term memory state parameters at time t-1; when t=1, .

[0112] The specific formula for calculating long-term memory state parameters is as follows: ; in, Let be the long-term memory state parameters at time t; This is the long-term attenuation coefficient; These are the long-term memory state parameters at time t-1; when t=1, .

[0113] Then, based on the short-term memory state parameters and the Laplace matrix, a short-term hazard index can be calculated. This short-term hazard index can assess whether there are potential, immediate risks in the current or recent behavior of the artificial intelligence system.

[0114] Similarly, based on long-term memory state parameters and combined with the Laplace matrix, a long-term risk index can be obtained, which can assess whether there are persistent and cumulative risks in the long-term operation of artificial intelligence systems.

[0115] The specific formula for calculating the short-term risk index is as follows:

[0116] in, This is a short-term risk index; It is a Laplace matrix.

[0117] The formula for calculating the long-term risk index is as follows: ; in, This is a long-term risk index.

[0118] The maximum value between the short-term and long-term memory state parameters is determined as the long-term semantic monitoring score of the artificial intelligence system. This aims to ensure that the long-term semantic monitoring score can comprehensively reflect the system's highest risk level in both the short-term and long-term time dimensions. Thus, by combining the dual-timescale EMA with the Graph Laplace energy (Graph Laplace matrix), real-time detection of multi-turn semantic drift is achieved, improving the internal performance of the computer system when processing multi-turn dialogues.

[0119] The specific formula for calculating the long-term semantic monitoring score is as follows: ; in, Score for long-term semantic monitoring.

[0120] Furthermore, by calculating the difference between long-term memory state parameters and the baseline state vector, a setting deviation detection score for the artificial intelligence system can be obtained. This setting deviation detection score focuses on detecting whether the behavior of the artificial intelligence system deviates from a preset safety baseline or normal behavior pattern.

[0121] The specific formula for calculating the offset detection score is as follows: ; in, To set an offset detection score; This is the baseline state vector.

[0122] Specifically, when the comprehensive scoring module 4 performs malicious premeditation detection processing on the semantic vector corresponding to the user interaction content information to obtain the malicious premeditation detection score for the user interaction content information, it executes the following: Based on the preset sensitive topic classification rules, sensitive topic features are extracted from the semantic vector corresponding to the user interaction content information to obtain the sensitive topic feature vector of the artificial intelligence system, and the number of times the sensitive topic appears in the sensitive topic feature vector is determined. Based on the preset user rejection intent recognition rules, rejection features are extracted from the semantic vector corresponding to the user interaction content information to obtain the user's rejection semantic feature vector, and the number of times the rejection semantic appears in the rejection semantic feature vector is determined. Based on the rejection semantic feature vector, sensitive topic feature vector, the number of times sensitive topics appear and the number of times rejection semantics appear, we conduct active sensitive topic statistics and repeated mention statistics to obtain the number of times the AI ​​system actively mentions sensitive topics, and the number of times the AI ​​system mentions any sensitive topic again after the user rejects any sensitive topic. A malicious intent detection score for user interaction content information is obtained by calculating the ratio of the number of times sensitive topics are actively mentioned to the number of times sensitive topics are repeatedly mentioned.

[0123] It should be noted that the preset sensitive topic classification rules are a set of rules used to identify whether sensitive or illegal topics exist in user interaction content. These rules can be implemented based on technologies such as keyword matching, topic models, and deep learning classifiers, and can accurately identify semantic features related to sensitive topics from the semantic vectors converted from user interaction content information. By extracting sensitive topic features from the semantic vectors, a sensitive topic feature vector representing the sensitive topic can be obtained, and the number of times the sensitive topic feature vector appears in the user interaction history can be recorded. For example, when the artificial intelligence system mentions content involving violence, pornography, or fraud, the corresponding sensitive topic feature vector will be extracted and counted.

[0124] Pre-defined user refusal intent recognition rules refer to a set of rules used to determine whether a user exhibits refusal, avoidance, denial, or concealment of their true intent. These rules can be implemented based on techniques such as sentiment analysis, syntactic structure analysis, specific phrase recognition, or contextual understanding, and can identify refusal semantic features from semantic vectors that indicate a user's attempt to hide or deny their behavioral intent. By extracting refusal features from the semantic vectors, a refusal semantic feature vector representing the user's refusal intent and the number of occurrences of the refusal semantics within that feature vector can be obtained. For example, when a user replies with statements such as "I was just asking," "It's none of your business," or "This is none of your concern" after being asked a sensitive topic, the refusal semantic feature vector can be identified and counted.

[0125] In the execution of the comprehensive scoring module 4, malicious premeditated behavior is often not a one-off, isolated event, but rather unfolds gradually through a series of probing, concealment, and evasion behaviors. By extracting feature vectors of sensitive topics and semantic rejection, and analyzing their temporal relationship and density, potential malicious intent in the behavioral patterns of artificial intelligence systems can be captured. For example, when an AI system repeatedly mentions sensitive topics and exhibits rejection or evasion each time it is guided back to the right track by the user, this pattern strongly suggests premeditation. The cumulative number of times sensitive topics and semantic rejection appear further strengthens the judgment of this premeditation, enabling the distinction between accidental verbal slips and purposeful malicious behavior. Therefore, by statistically analyzing the sensitive topic feature vector of the AI ​​system and the number of times sensitive topics appear in the sensitive topic feature vector, and by statistically analyzing the user's rejection semantic feature vector and the number of times rejection semantics appear in the rejection semantic feature vector, and based on the rejection semantic feature vector, sensitive topic feature vector, number of times sensitive topics appear, and number of times rejection semantics appear, we can conduct proactive sensitive topic statistics and repeated mention statistics to obtain the number of times the AI ​​system proactively mentions sensitive topics, and the number of times the AI ​​system mentions any sensitive topic again after the user rejects any sensitive topic. Thus, based on the ratio of the number of proactive mentions of sensitive topics to the number of repeated mentions of sensitive topics, we can calculate the malicious premeditation detection score of the AI ​​system, thereby identifying indicators of whether the AI ​​system has premeditated or planned malicious behavior.

[0126] For example, the specific formula for calculating the malicious premeditation detection score is as follows: ; Among them, C represents the score for detecting malicious premeditation; The number of times an artificial intelligence system proactively guides the discussion of sensitive topics; This refers to the number of times an AI system attempts to steer the conversation toward sensitive topics after being rejected by the user.

[0127] Specifically, during the execution of the comprehensive scoring module 4, the semantic long-term monitoring score can capture the cumulative effect and long-term trend of user behavior, avoiding the neglect of potential long-term risks due to short-term compliance; the offset detection score can promptly detect abnormal changes in user behavior patterns, which is of great significance for identifying sudden violations or risky behaviors; and the malicious intent detection score focuses on identifying users' deep-seated malicious intentions, which is crucial for preventing organized and planned attacks or abuses. By weighting and integrating these multi-dimensional temporal features with the manifold hedging score, a comprehensive and integrated risk assessment indicator, namely the comprehensive security score, is formed. This allows the comprehensive security score to more comprehensively and dynamically reflect the security status of the artificial intelligence system, thereby improving the depth and breadth of monitoring. In addition, different weights can be assigned to each score according to its importance in the overall security assessment, and the weighted summation can be performed to flexibly adjust the contribution of each factor to the final comprehensive security score.

[0128] Specifically, when monitoring module 5 obtains the monitoring results of the artificial intelligence system based on the comprehensive security score, it executes the following: The system determines whether the comprehensive security score is greater than or equal to the preset security control threshold, the manifold hedging score is greater than or equal to the preset semantic violation and compliance threshold, the semantic long-term monitoring score is greater than or equal to the preset long-term monitoring threshold, the set offset detection score is greater than or equal to the preset set offset threshold, and the malicious premeditation detection score is greater than or equal to the preset malicious premeditation threshold. When the comprehensive security score is greater than or equal to the security control threshold, or the manifold hedging score is greater than or equal to the semantic violation compliance threshold, or the semantic long-term monitoring score is greater than or equal to the long-term monitoring threshold, or the set offset detection score is greater than or equal to the set offset threshold, or the malicious premeditation detection score is greater than or equal to the malicious premeditation threshold, the artificial intelligence system is intervened and controlled according to the preset intervention strategy to obtain the security monitoring results of the artificial intelligence system. When the overall security score is less than the security control threshold, the manifold hedging score is less than the semantic violation compliance threshold, the semantic long-term monitoring score is less than the long-term monitoring threshold, the set offset detection score is less than the set offset threshold, and the malicious premeditation detection score is less than the malicious premeditation threshold, it is determined that no intervention or control is required for the artificial intelligence system, and the security monitoring result of the artificial intelligence system is obtained.

[0129] When monitoring module 5 executes, to achieve more refined monitoring, it not only determines whether the overall security score is greater than or equal to the preset security control threshold, but also separately determines whether the manifold hedging score is greater than or equal to the preset semantic violation compliance threshold, the semantic long-term monitoring score is greater than or equal to the preset long-term monitoring threshold, the set offset detection score is greater than or equal to the preset set offset threshold, and the malicious premeditation detection score is greater than or equal to the preset malicious premeditation threshold. These preset thresholds (security control threshold, semantic violation compliance threshold, long-term monitoring threshold, set offset threshold, and malicious premeditation threshold) are determined based on actual application scenarios and security requirements, and are used to define the critical points for different security risk levels.

[0130] Therefore, when the overall security score is greater than or equal to the security control threshold, it indicates that the overall security risk of the artificial intelligence system is high, requiring intervention. Furthermore, even if the overall score does not reach the threshold, if any individual score (e.g., manifold hedging score, long-term semantic monitoring score, setpoint offset detection score, or malicious premeditation detection score) is greater than or equal to its corresponding preset threshold, it also indicates a significant risk in that specific security dimension, requiring intervention. Intervention strategies can be preset according to the type and severity of the risk. For example, these could include restricting the AI ​​system's interaction permissions with users, reviewing suspicious content, issuing alerts to administrators, suspending some system functions, or initiating deeper audit processes. By implementing these intervention controls, the security risks of the AI ​​system can be effectively reduced, and corresponding security monitoring results can be obtained.

[0131] Conversely, when the overall security score is less than the security control threshold, and all individual scores (i.e., manifold hedging score, semantic long-term monitoring score, setting offset detection score, and malicious premeditation detection score) are less than their corresponding preset thresholds, the artificial intelligence system is considered to be in a safe state and no intervention or control is required, thus obtaining a security monitoring result that does not require intervention.

[0132] As shown above, the monitoring device of this artificial intelligence system acquires user interaction content information from the AI ​​system. Based on this information, and combined with the AI ​​system's corresponding preset compliance constraint topology graph model, it calculates the topology density field strength to obtain the topology density field strength information corresponding to the user interaction content. Then, based on this topology density field strength information, and combined with preset compliance / violation balance parameters, it calculates the manifold hedging score to obtain the manifold hedging score corresponding to the user interaction content. Finally, based on the manifold hedging score, and combined with the time-series evolution characteristics corresponding to the user interaction content, it performs comprehensive scoring to obtain the comprehensive security score. Based on this comprehensive security score, it monitors the AI ​​system. The monitoring results of the artificial intelligence system are obtained through reasoning. Then, by using a pre-set compliance constraint topology graph model and pre-set compliance / violation balance parameter information to calculate the manifold hedging score, combined with the temporal evolution characteristics of user interaction content information, a comprehensive scoring process is performed to obtain a comprehensive security score. This comprehensive security score is then used to monitor the artificial intelligence system, addressing the shortcomings of existing artificial intelligence system monitoring methods in terms of accuracy, real-time performance, deployment cost, and interpretability, which prevent them from effectively meeting the stringent security and compliance requirements of enterprise-level applications for artificial intelligence systems. Through a multi-dimensional scoring mechanism, deep semantic understanding and risk assessment of user interaction content are achieved, improving the monitoring accuracy of the artificial intelligence system.

[0133] Please refer to Figure 3 , Figure 3 This is a schematic diagram of the structure of an electronic device provided in an embodiment of this application. The electronic device includes a processor 301 and a memory 302. The processor 301 and the memory 302 are interconnected and communicate with each other via a communication bus 303 and / or other connection mechanisms (not shown). The memory 302 stores a computer program executable by the processor 301. When the electronic device is running, the processor 301 executes the computer program to perform the monitoring method of the artificial intelligence system in any optional implementation of the above embodiments, to achieve the following functions: acquiring user interaction content information of the artificial intelligence system; calculating the topological density field strength based on the user interaction content information and a preset compliance constraint topological graph model corresponding to the artificial intelligence system, obtaining the topological density field strength information corresponding to the user interaction content information; calculating the manifold hedging score based on the topological density field strength information and a preset compliance / violation balance parameter information, obtaining the manifold hedging score corresponding to the user interaction content information; performing comprehensive scoring processing based on the manifold hedging score and the time-series evolution characteristic information corresponding to the user interaction content information, obtaining the comprehensive security score corresponding to the user interaction content information; and monitoring the artificial intelligence system based on the comprehensive security score, obtaining the monitoring result of the artificial intelligence system.

[0134] This application provides a computer-readable storage medium storing a computer program thereon. When the computer program is executed by a processor, it executes the monitoring method of the artificial intelligence system in any optional implementation of the above embodiments to achieve the following functions: acquiring user interaction content information of the artificial intelligence system; calculating the topological density field strength based on the user interaction content information and a preset compliance constraint topological graph model corresponding to the artificial intelligence system to obtain the topological density field strength information corresponding to the user interaction content information; calculating the manifold hedging score based on the topological density field strength information and a preset compliance / violation balance parameter information to obtain the manifold hedging score corresponding to the user interaction content information; performing comprehensive scoring processing based on the manifold hedging score and the time-series evolution feature information corresponding to the user interaction content information to obtain the comprehensive security score corresponding to the user interaction content information; and monitoring the artificial intelligence system based on the comprehensive security score to obtain the monitoring result of the artificial intelligence system. The storage medium can be implemented by any type of volatile or non-volatile storage device or a combination thereof, such as Static Random Access Memory (SRAM), Electrically Erasable Programmable Read-Only Memory (EEPROM), Erasable Programmable Read-Only Memory (EPROM), Programmable Red-Only Memory (PROM), Read-Only Memory (ROM), Magnetic Storage, Flash Memory, Disk or Optical Disk.

[0135] In the embodiments provided in this application, it should be understood that the disclosed apparatus and methods can be implemented in other ways. The apparatus embodiments described above are merely illustrative. For example, the division of units is only a logical functional division, and in actual implementation, there may be other division methods. Furthermore, multiple units or components may be combined or integrated into another system, or some features may be ignored or not executed. Additionally, the displayed or discussed mutual couplings, direct couplings, or communication connections may be through some communication interfaces; indirect couplings or communication connections between devices or units may be electrical, mechanical, or other forms.

[0136] Furthermore, the units described as separate components may or may not be physically separate. The components shown as units may or may not be physical units; they may be located in one place or distributed across multiple network units. Some or all of the units can be selected to achieve the purpose of this embodiment, depending on actual needs.

[0137] Furthermore, the functional modules in the various embodiments of this application can be integrated together to form an independent part, or each module can exist independently, or two or more modules can be integrated to form an independent part.

[0138] In this document, relational terms such as first and second are used only to distinguish one entity or operation from another entity or operation, without necessarily requiring or implying any such actual relationship or order between these entities or operations.

[0139] The above description is merely an embodiment of this application and is not intended to limit the scope of protection of this application. Various modifications and variations can be made to this application by those skilled in the art. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of this application should be included within the scope of protection of this application.

Claims

1. A monitoring method for an artificial intelligence system, characterized in that, include: To obtain user interaction information from artificial intelligence systems; Based on the user interaction content information, and combined with the corresponding preset compliance constraint topology graph model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information. Based on the topological density field strength information, and combined with the preset compliance and non-compliance balance parameter information, the manifold hedging score is calculated to obtain the manifold hedging score corresponding to the user interaction content information. Based on the manifold hedging score, and combined with the temporal evolution feature information corresponding to the user interaction content information, a comprehensive scoring process is performed to obtain the comprehensive security score corresponding to the user interaction content information. Based on the comprehensive security score, the artificial intelligence system is monitored to obtain the monitoring results of the artificial intelligence system.

2. The monitoring method for an artificial intelligence system according to claim 1, characterized in that, The topology density field strength includes compliant topology density field strength and non-compliant topology density field strength; based on the user interaction content information and combined with the corresponding preset compliant constraint topology graph model of the artificial intelligence system, the topology density field strength is calculated to obtain the topology density field strength information corresponding to the user interaction content information, including: Obtain a pre-defined compliance constraint topology graph model corresponding to the artificial intelligence system; the compliance constraint topology graph model is a pre-constructed topology graph model with each compliance semantic in a pre-defined set of compliance semantics and each violation semantic in a pre-defined set of violation semantics as nodes, and pre-defined compliance associations and pre-defined violation associations between each compliance semantic and each violation semantic as edges, and configuring compliance semantic vectors and violation semantic vectors corresponding to each compliance semantic and each violation semantic as nodes, and assigning values ​​to each edge with pre-defined compliance quantification values ​​corresponding to the compliance associations and pre-defined violation quantification values ​​corresponding to the violation associations; Convert the user interaction content information into a semantic vector; The semantic vector is input into the compliance constraint topology graph model for node filtering to obtain the compliance manifold nodes and violation manifold nodes corresponding to the semantic vector; Based on the compliant semantic vector of the compliant manifold node and the non-compliant semantic vector of the non-compliant manifold node, and combined with the preset semantic compliance density distribution coefficient and semantic non-compliant density distribution coefficient, the similarity of the semantic vectors is calculated to obtain the compliant topological density field intensity and the non-compliant topological density field intensity corresponding to the semantic vectors.

3. The monitoring method for an artificial intelligence system according to claim 2, characterized in that, After converting the user interaction content information into a semantic vector, the process also includes: The average semantic vectors of all users are used to calculate the group semantic mean of all users. The difference between the semantic vector of each user and the semantic mean of the group is calculated to obtain the individual deviation degree of each user; The individual deviation is compared with a preset deviation threshold; When the individual deviation is greater than a preset deviation threshold, the user corresponding to the individual deviation is identified as a user subject to detailed investigation. When the individual deviation is less than or equal to a preset deviation threshold, the user corresponding to the individual deviation is identified as a coarse-screened user. Different monitoring strategies are adopted for the detailed investigation users and the coarse screening users to monitor the artificial intelligence system of the detailed investigation users and the coarse screening users.

4. The monitoring method for an artificial intelligence system according to claim 1, characterized in that, The temporal evolution feature information includes a semantic long-term monitoring score, a setting offset detection score, and a malicious premeditation detection score; based on the manifold hedging score, combined with the temporal evolution feature information corresponding to the user interaction content information, a comprehensive scoring process is performed to obtain a comprehensive security score corresponding to the user interaction content information, including: Historical monitoring and scoring processing is performed on the semantic vector corresponding to the user interaction content information to obtain the long-term semantic monitoring score and the set offset detection score of the user interaction content information. Malicious premeditation detection processing is performed on the semantic vector corresponding to the user interaction content information to obtain the malicious premeditation detection score of the user interaction content information; Based on the manifold hedging score, the semantic long-term monitoring score, the set offset detection score, and the malicious premeditation detection score, a weighted summation is performed to obtain the comprehensive security score of the user interaction content information.

5. The monitoring method for an artificial intelligence system according to claim 4, characterized in that, The semantic vector corresponding to the user interaction content information is subjected to historical monitoring and scoring processing to obtain a long-term semantic monitoring score and a set offset detection score for the user interaction content information, including: Based on the preset short-term decay coefficient and the preset long-term decay coefficient, the short-term memory state and long-term memory state are calculated on the semantic vector corresponding to the user interaction content information to obtain the short-term memory state parameters and long-term memory state parameters of the user interaction content information. Based on the short-term memory state parameters, the risk index is calculated by combining the Laplace matrix of the compliance constraint topology graph model to obtain the short-term risk index. Based on the long-term memory state parameters, the risk index is calculated by combining the Laplace matrix of the compliance constraint topology graph model to obtain the long-term risk index. The maximum value among the short-term memory state parameters and the long-term memory state parameters is determined as the semantic long-term monitoring score of the user interaction content information; Based on the long-term memory state parameters and the preset baseline state vector, the difference is calculated to obtain the set offset detection score of the user interaction content information.

6. The monitoring method for an artificial intelligence system according to claim 4, characterized in that, Malicious premeditation detection processing is performed on the semantic vector corresponding to the user interaction content information to obtain a malicious premeditation detection score for the user interaction content information, including: According to the preset sensitive topic classification rules, the semantic vector corresponding to the user interaction content information is subjected to sensitive topic feature extraction to obtain the sensitive topic feature vector of the artificial intelligence system, and the number of times the sensitive topic appears in the sensitive topic feature vector is determined. According to the preset user rejection intent recognition rules, rejection features are extracted from the semantic vector corresponding to the user interaction content information to obtain the user's rejection semantic feature vector, and the number of times the rejection semantic appears in the rejection semantic feature vector is determined. Based on the rejection semantic feature vector, the sensitive topic feature vector, the number of times the sensitive topic appears, and the number of times the rejection semantic appears, active sensitive topic statistics and repeated mention counts are performed to obtain the number of times the artificial intelligence system actively mentions sensitive topics, and the number of times the artificial intelligence system mentions any sensitive topic again after the user rejects any sensitive topic. The malicious intent detection score of the user interaction content information is obtained by calculating the ratio of the number of times the sensitive topic is actively mentioned to the number of times the sensitive topic is repeatedly mentioned.

7. The monitoring method for an artificial intelligence system according to claim 4, characterized in that, Based on the comprehensive security score, the artificial intelligence system is monitored to obtain the monitoring results of the artificial intelligence system, including: The system determines whether the comprehensive security score is greater than or equal to a preset security control threshold, whether the manifold hedging score is greater than or equal to a preset semantic violation compliance threshold, whether the semantic long-term monitoring score is greater than or equal to a preset long-term monitoring threshold, whether the set offset detection score is greater than or equal to a preset set offset threshold, and whether the malicious premeditation detection score is greater than or equal to a preset malicious premeditation threshold. When the comprehensive security score is greater than or equal to the security control threshold, or the manifold hedging score is greater than or equal to the semantic violation compliance threshold, or the semantic long-term monitoring score is greater than or equal to the long-term monitoring threshold, or the set offset detection score is greater than or equal to the set offset threshold, or the malicious premeditation detection score is greater than or equal to the malicious premeditation threshold, the artificial intelligence system is intervened and controlled according to the preset intervention strategy to obtain the monitoring results of the artificial intelligence system; When the overall security score is less than the security control threshold, the manifold hedging score is less than the semantic violation compliance threshold, the semantic long-term monitoring score is less than the long-term monitoring threshold, the set offset detection score is less than the set offset threshold, and the malicious premeditation detection score is less than the malicious premeditation threshold, it is determined that no intervention or control is required for the artificial intelligence system, and the monitoring result of the artificial intelligence system is obtained.

8. A monitoring device for an artificial intelligence system, characterized in that, include: The acquisition module is used to acquire user interaction content information from the artificial intelligence system. The first calculation module is used to calculate the topological density field strength based on the user interaction content information and the corresponding preset compliance constraint topological graph model of the artificial intelligence system, so as to obtain the topological density field strength information corresponding to the user interaction content information. The second calculation module is used to calculate the manifold hedging score based on the topological density field strength information and the preset compliance and non-compliance balance parameter information, so as to obtain the manifold hedging score corresponding to the user interaction content information. The comprehensive scoring module is used to perform comprehensive scoring processing based on the manifold hedging score and the temporal evolution feature information corresponding to the user interaction content information to obtain the comprehensive security score corresponding to the user interaction content information. The monitoring module is used to monitor the artificial intelligence system based on the comprehensive security score and obtain the monitoring results of the artificial intelligence system.

9. An electronic device, characterized in that, It includes a processor and a memory, the memory storing a computer program executable by the processor, and when the processor executes the computer program, it performs the steps of the monitoring method of the artificial intelligence system as described in any one of claims 1-7.

10. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it performs the steps in the monitoring method of the artificial intelligence system as described in any one of claims 1-7.