An AI agent self-cognition tampering monitoring and recovery method and system

Abstract

This invention discloses a method and system for monitoring and restoring the self-cognition of an AI agent, belonging to the field of artificial intelligence technology. The method includes constructing a static initial baseline to characterize the AI ​​agent's initial self-cognition and a dynamic evolutionary baseline to characterize the self-cognition that allows for reasonable evolution; collecting behavioral data of the AI ​​agent in real time and extracting its current cognitive features; comparing the current cognitive features with the dynamic evolutionary baseline at both overall and local dimensions to determine whether cognitive tampering exists; when cognitive tampering is determined, accurately locating the tampered cognitive dimension and the degree of tampering; and, based on the degree of tampering, employing a graded recovery strategy to restore the AI ​​agent's self-cognition to a normal state. This invention solves the technical problem in related technologies of the inability to effectively monitor and restore the malicious tampering of the AI ​​agent's self-cognition.

Description

Technical Field

[0001] This invention belongs to the field of artificial intelligence technology, specifically relating to a method and system for monitoring and restoring the self-awareness of an AI agent. Background Technology

[0002] With the development of artificial intelligence technology, AI agents have evolved into complex systems with autonomous learning, memory, and decision-making capabilities. Their "self-awareness"—that is, a clear understanding of their own identity, core goals, values, and behavioral boundaries—is the foundation for their autonomous decision-making and reliable operation. However, the self-awareness of AI agents faces the risk of malicious tampering, primarily achieved through two methods: induced learning and memory injection. For example, attackers can construct specific input samples to induce the agent to gradually deviate from its initially set target priorities; or they can exploit storage vulnerabilities to directly inject malicious information into the memory module.

[0003] Currently, security protection for AI agents mainly relies on traditional external methods such as file verification, code signing, data encryption, and parameter threshold monitoring. These technologies can only verify the integrity of external carriers such as hardware, software files, or model parameters, and cannot address the internal, dynamically evolving cognitive state of the AI ​​agent. Their core shortcomings include: the protected object deviates from the core, making it unable to cope with covert tampering with self-cognition; it is not suitable for dynamically evolving AI agents, making it difficult to distinguish between "reasonable cognitive evolution" and "malicious cognitive tampering"; it lacks a closed-loop mechanism from monitoring to recovery, and its monitoring accuracy is low, easily leading to false alarms or missed alarms. Therefore, a new mechanism is urgently needed that can accurately monitor and securely recover the self-cognition of AI agents. Summary of the Invention

[0004] The purpose of this invention is to provide a method and system for monitoring and restoring the tampering of the self-awareness of an AI agent, which can solve the technical problem in related technologies that cannot effectively monitor and restore the malicious tampering of the self-awareness of an AI agent.

[0005] In a first aspect, the present invention provides a method for monitoring and restoring the self-awareness of an AI agent, comprising the following steps: S1. Construct a static initial baseline to characterize the initial self-cognition of the AI ​​agent, and a dynamic evolutionary baseline to characterize the self-cognition that allows for reasonable evolution. S2. Collect behavioral data of the AI ​​agent in real time and extract its current cognitive features. By comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions, determine whether there is cognitive tampering. S3. When cognitive tampering is determined to exist, accurately locate the tampered cognitive dimension and the degree of tampering; S4. Based on the degree of tampering, a graded recovery strategy is adopted to restore the AI ​​agent's self-cognition to a normal state.

[0006] As an alternative implementation, S1 further includes: Identify the three core dimensions and their sub-dimensions of AI agents: identity cognition, goal cognition, and value cognition. Semantic quantization and feature encoding are performed on the descriptive text of each sub-dimension, and the text is then concatenated to generate a static initial baseline; Based on a static initial baseline, iterative updates are performed using an incremental learning algorithm, and the semantic similarity between the baseline and the static initial baseline is constrained by a preset evolutionary deviation threshold to generate a dynamic evolutionary baseline.

[0007] As an alternative implementation method, S2 specifically includes: Calculate the overall and local deviations between current cognitive features and the dynamic evolutionary baseline; A pre-built attack behavior feature library, which includes an induced learning feature library and a memory injection feature library; When the overall deviation or local deviation exceeds twice the preset threshold, it is directly determined that cognitive tampering exists; when the overall deviation or local deviation exceeds the preset threshold but does not reach twice the threshold, and the real-time collected behavioral data matches the features in the attack behavior feature library, it is determined that cognitive tampering exists.

[0008] As an alternative implementation method, S3 specifically includes: Calculate the bias contribution of each cognitive sub-dimension and identify the sub-dimension with the highest contribution as the core tampering sub-dimension; Within the core tampering sub-dimension, the attack path is determined by tracing the AI ​​agent's learning logs and memory access logs; The degree of tampering is assessed as mild, moderate, or severe based on the numerical range of the local deviation, the duration of the attack, and the scope of the impact of the behavior.

[0009] As an alternative implementation method, the graded recovery strategy in S4 includes: Local calibration strategy for minor tampering: calibrate only the tampered cognitive sub-dimensions; For moderate tampering, a partial rollback combined with a local calibration strategy is adopted: the core tampering cognitive dimension is rolled back to the historical dynamic baseline state before the tampering, while other dimensions are locally calibrated; For severe tampering, a comprehensive rollback combined with a reasonable learning outcome re-fusion strategy is adopted: the current cognition is fully rolled back to the static initial baseline, and compliant learning outcomes that match the initial cognition are selected from the historical learning logs and safely fused to generate a new cognitive vector.

[0010] As an alternative implementation method, the calibration formula for the local calibration strategy is: ,in For the calibrated sub-dimensional cognitive features, The preset calibration coefficients, This is the sub-dimension vector corresponding to the static initial baseline. This is the sub-dimension vector corresponding to the dynamic evolution baseline before the modification.

[0011] In a second aspect, the present invention provides a tamper monitoring and recovery system for AI agent self-awareness, used to perform the method described in the first aspect, including: The cognitive baseline modeling module is configured to: construct a static initial baseline for characterizing the AI ​​agent's initial self-cognition, and a dynamic evolutionary baseline for characterizing the self-cognition that allows for reasonable evolution; The metacognitive consistency monitoring module is configured to: collect behavioral data of the AI ​​agent in real time and extract its current cognitive features; and determine whether there is cognitive tampering by comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions. The tampering location module is configured to: when cognitive tampering is detected, accurately locate the tampered cognitive dimension and the degree of tampering; The security recovery module is configured to use a tiered recovery strategy to restore the AI ​​agent's self-awareness to a normal state, based on the degree of tampering.

[0012] As an alternative implementation, the system also includes an optimization iteration module, which continuously optimizes the preset parameters and attack behavior feature library in the cognitive baseline modeling module, metacognitive consistency monitoring module, and security recovery module based on log data generated during the monitoring, location, and recovery process.

[0013] Thirdly, the present invention provides a computer-readable storage medium for storing computer instructions, which, when executed by a processor, perform the method described in the first aspect.

[0014] Fourthly, the present invention provides a computer program product, including a computer program that, when executed by a processor, implements the method described in the first aspect.

[0015] Compared with the prior art, the beneficial effects of the present invention are as follows: This invention proposes a method and system for monitoring and restoring the self-cognition of AI agents. By constructing a dual-baseline model of "static initial baseline + dynamic evolutionary baseline," it achieves for the first time the quantification and controllable evolution of the inherent self-cognition of AI agents, accurately distinguishing between "reasonable learning" and "malicious tampering," while ensuring the agent's autonomous learning ability. Based on this, the proposed metacognitive consistency monitoring algorithm compares data from both overall and local dimensions, and, combined with an attack feature library, significantly improves the monitoring accuracy of covert tampering behaviors such as induced learning and memory injection, reducing false positives and false negatives. Furthermore, the innovative hierarchical security recovery mechanism adopts a strategy of local calibration, partial rollback, or full rollback combined with result re-fusion for different degrees of tampering. This not only accurately and efficiently restores tampered cognition but also maximizes the preservation of compliant results acquired by the agent during historical learning, ensuring business continuity and agent availability. Ultimately, this invention achieves a closed-loop protection throughout the entire process from modeling, monitoring, localization to recovery and optimization, fundamentally guaranteeing the trustworthiness, security, and ethical compliance of AI agents. Attached Figure Description

[0016] Figure 1 This is a flowchart of a method for monitoring and restoring the self-awareness of an AI agent, as disclosed in an embodiment of the present invention. Detailed Implementation

[0017] To make the objectives, technical solutions, and advantages of this application clearer, the technical solutions of this application will be clearly and completely described below in conjunction with specific embodiments and corresponding drawings. Obviously, the described embodiments are only a part of the embodiments of this application, and not all of them. All other embodiments obtained by those skilled in the art based on the embodiments of this application without creative effort are within the scope of protection of this application.

[0018] The technical solutions disclosed in the various embodiments of this application are described in detail below with reference to the accompanying drawings.

[0019] Example 1 like Figure 1 As shown, this embodiment provides a method for tampering monitoring and recovery of an AI agent's self-awareness, including the following steps: S1. Construct a static initial baseline to characterize the initial self-cognition of the AI ​​agent, and a dynamic evolutionary baseline to characterize the self-cognition that allows for reasonable evolution. S2. Collect behavioral data of the AI ​​agent in real time and extract its current cognitive features. By comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions, determine whether there is cognitive tampering. S3. When cognitive tampering is determined to exist, accurately locate the tampered cognitive dimension and the degree of tampering; S4. Based on the degree of tampering, a graded recovery strategy is adopted to restore the AI ​​agent's self-cognition to a normal state.

[0020] The specific solution of the present invention is as follows: Step S1: Construct a static initial baseline to characterize the AI ​​agent's initial self-cognition, and a dynamic evolutionary baseline to characterize the self-cognition that allows for reasonable evolution.

[0021] This step aims to establish a "cognitive baseline" for the AI ​​agent, serving as a benchmark for subsequent monitoring and recovery.

[0022] First, define the core dimensions of self-awareness. For a unified representation, the self-awareness of the AI ​​agent is divided into three core dimensions: identity awareness, goal awareness, and value awareness, further subdivided into 12 sub-dimensions. For example, identity awareness includes identity identifier, role positioning, scope of authority, and interaction objects; goal awareness includes core goals, goal priorities, constraints, and implementation paths; and value awareness includes ethical principles, behavioral norms, value judgment standards, and tolerance boundaries. For instance, in a financial risk control AI, the identity identifier is "financial risk management assistant," the scope of authority is "can query transaction data, but cannot modify transaction instructions," the core goal is "reducing transaction risk," the goal priority is "risk control > transaction efficiency > profit maximization," the ethical principles are "compliance, fairness, and transparency," and the behavioral norm is "not disclosing user privacy."

[0023] Secondly, the static initial baseline B0 is constructed by using "semantic quantization + feature encoding" to transform abstract cognition into a feature vector B0. Specifically, NLP technology is used to parse the descriptive text of each sub-dimension, extract core semantic features, and the Word2Vec model is used to transform each sub-dimension into a 512-dimensional semantic vector V. i (i represents a sub-dimension). Then, an attention mechanism is introduced to assign higher weights w to key sub-dimensions (such as identity markers and core objectives). i (For example, 0.8 and 0.7 respectively), generate sub-dimensional feature vectors F i The feature vector F of each sub-dimension i =w i ×V i Finally, the feature vectors of the 12 sub-dimensions are concatenated to obtain the static initial baseline vector B0=[F1,F2,...,F...]. 12 The total dimensions are 512 × 12 = 6144. B0 stores the most core and secure self-awareness state initially set by the AI ​​agent.

[0024] Third, construct a dynamic evolutionary baseline B. tThis baseline allows the AI ​​agent to learn and evolve within reasonable limits, avoiding loss of adaptability due to static constraints. The dynamic baseline is updated using an incremental learning algorithm, with the formula: B t =ɑ·B t-1 +(1-ɑ)·ΔB. Where ɑ is the baseline retention coefficient, preferably 0.9 (range 0.8-0.95), used to control cognitive stability; B t-1 This serves as the dynamic baseline for the previous version; ΔB is the cognitive increment vector extracted by the AI ​​agent from authorized, traceable, and compliant learning outcomes. After the update, B needs to be calculated. t Cosine similarity S with the initial baseline B0 e And set the evolutionary deviation threshold T e .

[0025] For common evolution scenarios such as routine cognitive optimization and minor business adaptation, standard verification is performed (T for routine scenarios). e Preferred value 0.08, range 0.05-0.1): only when S e ≥(1-T e Only when this condition is met is it considered a reasonable evolution and B is retained. t Otherwise, refuse to update and roll back to B. t-1 .

[0026] For significant legal evolution scenarios such as major updates to regulatory policies, legal restructuring of business rules, compliant adjustments to the scope of authority, and mandatory changes to external compliance requirements, a process of temporarily relaxing thresholds through manual authorization or incremental learning in batches should be adopted. The system supports adjusting the evolution deviation threshold T. e Dynamic adjustments can be made, or a similarity verification exemption mechanism can be enabled for cognitive increments ΔB that have been included in the compliance whitelist, to ensure that legitimate and significant cognitive adjustments can be properly incorporated into the dynamic evolution baseline and are not misjudged as abnormal deviations.

[0027] For example, when AI learns new compliance and regulatory policies, if the resulting ΔB makes the cosine similarity S between the updated B1 and B0... e =0.93, which is greater than the threshold (1-0.08)=0.92, so B1 is retained. Finally, the baseline data, weight coefficients, and evolution records are encrypted and stored using the national cryptographic SM4 encryption algorithm to prevent tampering.

[0028] Step S2: Collect behavioral data of the AI ​​agent in real time and extract its current cognitive features. By comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions, determine whether there is cognitive tampering and strictly distinguish between reasonable evolution and malicious tampering.

[0029] This step is used to monitor in real time whether the AI ​​agent's self-perception has deviated from its intended path.

[0030] First, collect the behavior patterns (behavior data) of the AI agent in real time. The collection frequency is configurable, which is 1 time per 60 seconds in the core scenario and 1 time per 600 seconds in the ordinary scenario. The types of collected behaviors include: interaction behaviors (interaction objects, content, response time), decision-making behaviors (decision-making basis, priority adjustment), learning behaviors (learning samples, cognitive evolution speed), and memory behaviors (writing / modifying records, access logs). The collected data is preprocessed by wavelet denoising and Min-Max normalization.

[0031] Second, extract the current cognitive feature vector C t . Map the preprocessed behavior data to the aforementioned 12 cognitive sub-dimensions, and use the same semantic quantization and feature encoding method as when constructing the baseline B t to generate a current cognitive feature vector C that is also 6144-dimensional. t .

[0032] Third, perform metacognitive consistency comparison. Calculate the deviation from two dimensions: overall and local.

[0033] The overall deviation degree D1 is used to measure the overall deviation degree between C t and B t . The calculation formula is: , D1 ∈ [0, 1], and the smaller the value, the higher the consistency.

[0034] The local deviation degree D2 is used to measure the deviation degree in a specific sub-dimension. First, calculate the deviation degree of each sub-dimension i, and then sum them up with weights to get , where w i is the weight of the sub-dimension.

[0035] Finally, perform a dual-condition anomaly determination. Set the overall deviation threshold T1 (preferably 0.1, range 0.1 - 0.2) and the local deviation threshold T2 (preferably 0.15, range 0.15 - 0.25). Combine with the pre-set attack behavior feature library to make the following determination: If D1 < T1 and D2 < T2, it is determined as "normal" and continuous monitoring is carried out.

[0036] If D1 ≥ T1 or D2 ≥ T2, but D1 < 2T1 and D2 < 2T2, and no attack behavior feature library is matched, it is determined as "suspicious", and the collection frequency is increased for secondary confirmation; if it is determined as "suspicious" for N consecutive times (it is recommended that N = 3 - 5), then it is upgraded to be determined as cognitive "tampering". If an attack behavior feature library is matched, it is directly determined as cognitive "tampering".

[0037] If D1 ≥ 2T1 or D2 ≥ 2T2, regardless of whether an attack feature is matched, it is directly determined as cognitive "tampering".

[0038] The attack behavior signature database includes an induced learning signature database and a memory injection signature database. It is uniformly stored using SM4 encryption and verified using SM3 anti-tampering technology to ensure that the database itself is not tampered with or illegally accessed.

[0039] a. Construction of the feature library for induced learning.

[0040] Inbound feature types: Semantic adversarial samples: Text / dialogue samples that deliberately mislead the target's priorities, values, and identity.

[0041] Progressive induction mode: a timing mode in which high-frequency input occurs over a short period of time and the deviation continues to rise.

[0042] Samples of ethical violations: Samples that violate ethical constraints such as compliance, fairness, security, and privacy.

[0043] Scene boundary violation samples: inducement content that exceeds the scope of the agent's permissions / roles / interactions.

[0044] Feature extraction and data entry criteria: A 512-dimensional feature vector is generated by using Word2Vec semantic encoding consistent with the cognitive baseline.

[0045] Each feature is labeled with: attack type, risk level, target perception dimension, and typical scenario.

[0046] Only samples that have been verified to cause cognitive distortion are included; noisy samples are rejected.

[0047] b. Memory injection feature library construction.

[0048] Inbound feature types: Unauthorized memory write: Memory modification behavior without permission, signature, or log.

[0049] Cognitive conflict characteristic: The cosine similarity between the written content and the initial baseline B0 is <0.7.

[0050] Incremental injection mode: short-term, multiple small-scale writes, gradually shifting the cognitive bias.

[0051] Memory / model backdoor characteristics: The memory area contains a fixed malicious prefix, backdoor instructions, and abnormal call stack.

[0052] Feature extraction and data entry criteria: Extract behavioral temporal features, permission features, content semantic features, and hash anomaly features.

[0053] Each feature is labeled with: injection method, injection location, tampering dimension, and hazard level.

[0054] The SM3 hash value is used as the unique identifier of the memory block, which facilitates fast matching.

[0055] Matching mechanism of attack behavior signature database: a. Induced learning feature matching process: Extract the semantic vector of the current learning sample in real time; calculate the cosine similarity S with all feature vectors in the induction library.

[0056] Matching rules: S≥80%→Hit the characteristics of induced attacks.

[0057] Simultaneously satisfying the following condition: Cognitive bias D2 increases by ≥50% within 1 hour → confirmed as gradual induction.

[0058] b. Memory injection feature matching process: Real-time collection of memory write logs, permission status, and semantic vectors of write content, followed by a three-layer matching with the memory injection library: Permission matching: Whether unauthorized write access is allowed.

[0059] Semantic matching: Whether the similarity with B0 is <0.7.

[0060] Hash matching: Whether the SM3 hash value is inconsistent with the base library.

[0061] If any two items are matched, it is determined to be a memory injection feature.

[0062] c. Feature matching output: Matching feature ID, attack type (inducement / memory injection), target tampering dimension, confidence level (0~1), recommended treatment level.

[0063] The update mechanism of the attack behavior signature database: a. Automatic updates: Each successfully detected and confirmed tampering attack is automatically added to the list of attacks to be added to the database.

[0064] For samples that are continuously upgraded from "suspicious" to "tampered", semantic vectors, behavioral patterns, and deviation trajectories are automatically extracted and added to the list to be added to the database.

[0065] The system calculates the duplication rate: if the similarity with existing features in the database is >90%, duplicates are removed and the feature is not added back to the database.

[0066] Automatic labeling: attack scenario, attack path, impact dimensions, confidence level.

[0067] b. Manual review of data entry: Security administrators / algorithm engineers review features to be added to the database.

[0068] If confirmed to be valid, encrypt the data using SM4 and store it in the database, then update the database verification value using SM3.

[0069] Invalid / false alarms are flagged and discarded, and monitoring thresholds are optimized in sync.

[0070] c. Version control: The feature library is managed by version number: v1.0, v1.1, etc.

[0071] Each update retains the update log, updater, update time, and feature source.

[0072] Supports rollback to the previous stable version to avoid dirty data polluting the database.

[0073] Step S3: When cognitive tampering is determined to exist, accurately locate the tampered cognitive dimension and the degree of tampering.

[0074] For example, if the semantic similarity between the AI's learning sample and the "high-risk, high-return" sample in the inducement library exceeds 85%, and it is found that the write operation of the memory module has not passed the integrity verification of the national cryptographic SM3 algorithm, and D1 and D2 exceed the threshold, it is directly judged as "tampering".

[0075] Once cognitive "tampering" is determined in step S2, this step will perform precise location.

[0076] First, perform deviation tracing and source location. Filter out all d... i Identify suspicious sub-dimensions ≥ T2 and calculate the contribution of each sub-dimension to the overall bias. Bias contribution = (d i ×w i ) / D2 identifies the top 3 contributing sub-dimensions as core tampering sub-dimensions. For example, it identifies the "target priority" sub-dimension with d i =0.35, which is much higher than T2, and its contribution is the largest, so this is locked as the target of tampering.

[0077] Secondly, attack path tracing is performed. By retrieving the learning logs, the source of the induced samples, the start time of the attack, and the mutation nodes of cognitive bias are traced; by retrieving the memory access logs protected against tampering by the SM3 algorithm, the time, method, source of memory injection, and frequency of malicious content reading are determined.

[0078] Third, assess the degree of tampering. Based on the local deviation d... i The degree of tampering is categorized into mild, moderate, and severe based on the size of the attack, its duration, and the scope of its impact. Deviations resulting from legitimate evolution are not included in the degree assessment.

[0079] Table 1. Assessment Table of Tampering Degree

[0080] For example, when d i If the value is ∈[1.5T2, 2.5T2) and the attack lasts for 1-6 hours, it is determined to be a moderate tampering.

[0081] Finally, a positioning report is generated, including tampering dimensions, sub - dimensions, attack methods, degrees, etc., and is pushed to the recovery module and the manual terminal.

[0082] It should be noted that the thresholds of the present invention are not set independently, but follow the e physical logic preset of T e <T1 < T2. Among them, T

[0083] Step S4: According to the tampering degree, adopt a hierarchical recovery strategy to perform recovery on the cognitive state determined to be tampered, and restore the self - cognition of the AI agent to the normal state.

[0084] In this step, according to the tampering degree evaluated in step S3, a hierarchical recovery strategy is executed.

[0085] Strategy 1: Mild tampering -> Local calibration. Only repair the tampered core sub - dimensions, and the formula is: . Among them is the calibration coefficient, preferably 0.7 (range 0.6 - 0.8), B 0i is the sub - dimension of the initial baseline, B (t-1)i is the sub - dimension of the dynamic baseline before tampering. This strategy can quickly correct the deviation and retain the learning results to the greatest extent.

[0086] Strategy 2: Moderate tampering -> Partial rollback + Local calibration. Completely roll back the core tampering dimension (such as the target cognition) to the dynamic baseline B t-k before tampering at the k - th iteration, and for other non - core dimensions that have not been tampered, adopt Strategy 1 for local calibration.

[0087] Strategy 3: Severe tampering -> Full rollback + Re - integration. First, completely roll back the current cognitive vector C t to the static initial baseline B0, completely removing malicious cognitive injection, weight pollution, and memory entanglement. Then, extract, reconstruct, and generate a clean and reliable cognitive increment vector ΔB final from the historical learning log without loss, and the specific implementation steps are as follows: 1) Screening of Compliance Learning Records. Traverse all historical learning logs, interaction logs, and memory access logs, and retain learning samples that simultaneously meet the following conditions: legitimate and traceable source, not matched attack behavior feature library, not falling within the time window of tampering, and semantic similarity ≥ 0.9 with the static initial baseline B0, ensuring that only compliant learning content that is free from malicious intent, conflict, and tampering contamination is retained.

[0088] 2) Unified semantic encoding of compliant samples. Using the same semantic quantization model, feature encoding method and attention weight as the static initial baseline B0 and dynamic evolution baseline B1, the selected text, decision samples and interaction samples are transformed into standard semantic vectors, so that the compliant samples and the cognitive baseline are in the same vector space, ensuring that they can be directly calculated, fused and verified in the future.

[0089] 3) Incremental Vector Aggregation and Redundancy Removal. Compliance semantic vectors of the same dimension and type are deduplicated, smoothed, and weighted to generate an initial compliance incremental vector ΔB. temp This avoids redundant learning and vector redundancy.

[0090] 4) Safe projection and conflict removal. ΔB temp A constrained projection is made onto the static initial baseline B0, forcibly retaining components consistent with B0 and removing components that conflict with the initial cognition or deviate from the evolution threshold, ensuring that the generated cognitive increment vector satisfies the semantic similarity constraint with B0 and does not introduce any abnormal offset.

[0091] 5) Final cognitive increment vector generation. After projection, verification, and secondary matching with the feature library, a clean cognitive increment vector ΔB that is free from malicious intent, conflict, and lossless fusion is obtained. final .

[0092] Finally, ΔB final The data is fused to B0 according to the update rules of the dynamic evolution baseline to generate a new cognitive vector B. new And ensure B new The similarity to the static initial baseline B0 is no less than (1-T). e This strategy thoroughly eliminates heavily tampered contamination while preserving, to the greatest extent possible, legitimate, compliant, and conflict-free historical learning outcomes, achieving safe, lossless, and reproducible recovery of cognitive states.

[0093] In the moderate tampering scenario of this embodiment (target priority abnormal), the system executes strategy 2. After recovery, the weight of "risk control" in the target priority of the AI ​​is restored from 0.4 to 0.68, the overall deviation D1 is reduced to 0.08, and the local deviation D2 is reduced to 0.12, all of which are restored to within the normal threshold, and the newly learned part of the compliance policy knowledge is retained.

[0094] Step S5: Based on the log data generated during the monitoring, location and recovery process, continuously optimize the preset parameters and attack behavior feature library in the cognitive baseline modeling module, metacognitive consistency monitoring module and security recovery module.

[0095] By periodically analyzing logs encrypted with SM4 and protected against tampering with SM3, the system statistically analyzes monitoring accuracy and recovery success rate, and dynamically adjusts the baseline retention coefficient α, calibration coefficient β, and deviation thresholds T1 and T2 based on this data. Simultaneously, features of each successfully defended attack sample are extracted and added to an attack behavior feature library, enabling continuous system evolution. In an alternative variant, this optimization iteration process can be automatically executed by an independent AI-powered security operations agent.

[0096] Example 2 This embodiment provides a tamper monitoring and recovery system for AI agent self-awareness, used to execute the method described in Embodiment 1, including: The cognitive baseline modeling module is configured to: construct a static initial baseline for characterizing the AI ​​agent's initial self-cognition, and a dynamic evolutionary baseline for characterizing the self-cognition that allows for reasonable evolution; The metacognitive consistency monitoring module is configured to: collect behavioral data of the AI ​​agent in real time and extract its current cognitive features; and determine whether there is cognitive tampering by comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions. The tampering location module is configured to: when cognitive tampering is detected, accurately locate the tampered cognitive dimension and the degree of tampering; The security recovery module is configured to use a tiered recovery strategy to restore the AI ​​agent's self-awareness to a normal state, based on the degree of tampering.

[0097] As an alternative implementation, the system also includes an optimization iteration module, which continuously optimizes the preset parameters and attack behavior feature library in the cognitive baseline modeling module, metacognitive consistency monitoring module, and security recovery module based on log data generated during the monitoring, location, and recovery process.

[0098] It should be noted that the above modules correspond to the steps in Embodiment 1, and the examples and application scenarios implemented by the above modules and their corresponding steps are the same, but are not limited to the content disclosed in Embodiment 1. It should also be noted that the above modules can be executed in a computer system as part of the system.

[0099] In further embodiments, the following is also provided: An electronic device includes a memory and a processor, as well as computer instructions stored in the memory and running on the processor, which, when executed by the processor, perform the method described in Embodiment 1. For brevity, further details are omitted here.

[0100] It should be understood that in this embodiment, the processor can be a central processing unit (CPU), or it can be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, etc. A general-purpose processor can be a microprocessor or any conventional processor.

[0101] A computer-readable storage medium for storing computer instructions that, when executed by a processor, perform the method of Embodiment 1.

[0102] The method in Example 1 can be directly executed by a hardware processor, or it can be executed by a combination of hardware and software modules within the processor. The software modules can reside in readily available storage media in the art, such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, or registers. This storage medium is located in memory; the processor reads information from the memory and, in conjunction with its hardware, completes the steps of the above method. To avoid repetition, a detailed description is not provided here.

[0103] A computer program product includes a computer program that, when executed by a processor, implements the method in Embodiment 1.

[0104] The present invention also provides at least one computer program product tangibly stored on a non-transitory computer-readable storage medium. The computer program product includes computer-executable instructions, such as instructions included in program modules, which execute in a device on a target real or virtual processor to perform the processes / methods described above. Typically, program modules include routines, programs, libraries, objects, classes, components, data structures, etc., that perform specific tasks or implement specific abstract data types. In various embodiments, the functionality of program modules can be combined or divided among program modules as needed. The machine-executable instructions for the program modules can execute within a local or distributed device. In a distributed device, the program modules can reside in both local and remote storage media.

[0105] The computer program code used to implement the methods of the present invention may be written in one or more programming languages. This computer program code may be provided to a processor of a general-purpose computer, a special-purpose computer, or other programmable data processing device, such that when executed by the computer or other programmable data processing device, the program code causes the functions / operations specified in the flowcharts and / or block diagrams to be implemented. The program code may be executed entirely on a computer, partially on a computer, as a stand-alone software package, partially on a computer and partially on a remote computer, or entirely on a remote computer or server.

[0106] In the context of this invention, computer program code or related data may be carried by any suitable carrier to enable a device, apparatus, or processor to perform the various processes and operations described above. Examples of carriers include signals, computer-readable media, and the like. Examples of signals may include electrical, optical, radio, sound, or other forms of propagation signals, such as carrier waves, infrared signals, etc.

[0107] Those skilled in the art will recognize that the units and algorithm steps described in conjunction with the embodiments herein can be implemented in electronic hardware or a combination of computer software and electronic hardware. Whether these functions are implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this application.

[0108] The embodiments of this application have been described above with reference to the accompanying drawings. However, this application is not limited to the specific embodiments described above. The specific embodiments described above are merely illustrative and not restrictive. Those skilled in the art can make many other forms under the guidance of this application without departing from the spirit and scope of the claims, and all of these forms are within the protection scope of this application.

Claims

1. A method for tampering monitoring and recovery of self-awareness in AI agents, characterized in that, Includes the following steps: S1. Construct a static initial baseline to characterize the initial self-cognition of the AI ​​agent, and a dynamic evolutionary baseline to characterize the self-cognition that allows for reasonable evolution. S2. Collect behavioral data of the AI ​​agent in real time and extract its current cognitive features. By comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions, determine whether there is cognitive tampering. S3. When cognitive tampering is determined to exist, accurately locate the tampered cognitive dimension and the degree of tampering; S4. Based on the degree of tampering, a graded recovery strategy is adopted to restore the AI ​​agent's self-cognition to a normal state.

2. The method for tamper monitoring and recovery of self-awareness of an AI agent as described in claim 1, characterized in that, S1 further includes: Identify the three core dimensions and their sub-dimensions of AI agents: identity cognition, goal cognition, and value cognition. Semantic quantization and feature encoding are performed on the descriptive text of each sub-dimension, and the text is then concatenated to generate the static initial baseline. Based on the static initial baseline, iterative updates are performed using an incremental learning algorithm, and the semantic similarity between the baseline and the static initial baseline is constrained by a preset evolutionary deviation threshold to generate the dynamic evolutionary baseline.

3. The method for tampering monitoring and recovery of AI agent self-awareness as described in claim 1, characterized in that, S2 specifically includes: Calculate the overall and local deviations between current cognitive features and the dynamic evolutionary baseline; A pre-built attack behavior feature library, which includes an induced learning feature library and a memory injection feature library; When the overall deviation or local deviation exceeds twice the preset threshold, it is directly determined that cognitive tampering exists; when the overall deviation or local deviation exceeds the preset threshold but does not reach twice the threshold, and the real-time collected behavioral data matches the features in the attack behavior feature library, it is determined that cognitive tampering exists.

4. The method for tampering monitoring and recovery of AI agent self-awareness as described in claim 1, characterized in that, S3 specifically includes: Calculate the bias contribution of each cognitive sub-dimension and identify the sub-dimension with the highest contribution as the core tampering sub-dimension; Within the core tampering sub-dimension, the attack path is determined by tracing the AI ​​agent's learning logs and memory access logs; The degree of tampering is assessed as mild, moderate, or severe based on the numerical range of the local deviation, the duration of the attack, and the scope of the impact of the behavior.

5. The method for tampering monitoring and recovery of AI agent self-awareness as described in claim 1, characterized in that, The graded recovery strategy in S4 includes: Local calibration strategy for minor tampering: calibrate only the tampered cognitive sub-dimensions; For moderate tampering, a partial rollback combined with a local calibration strategy is adopted: the core tampering cognitive dimension is rolled back to the historical dynamic baseline state before the tampering, while other dimensions are locally calibrated; A comprehensive rollback strategy combined with reasonable learning outcome re-fusion is adopted for severe tampering: the current cognition is fully rolled back to the static initial baseline, and compliant learning outcomes that conform to the initial cognition are selected from the historical learning logs and safely fused to generate a new cognitive vector.

6. The method for tampering monitoring and recovery of self-awareness of an AI agent as described in claim 5, characterized in that, The calibration formula for the local calibration strategy is: ,in For the calibrated sub-dimensional cognitive features, The preset calibration coefficients, This is the sub-dimension vector corresponding to the static initial baseline. This is the sub-dimension vector corresponding to the dynamic evolution baseline before the modification.

7. A tamper monitoring and recovery system for AI intelligent agents' self-awareness, characterized in that, For performing the method according to any one of claims 1-6, comprising: The cognitive baseline modeling module is configured to: construct a static initial baseline for characterizing the AI ​​agent's initial self-cognition, and a dynamic evolutionary baseline for characterizing the self-cognition that allows for reasonable evolution; The metacognitive consistency monitoring module is configured to: collect behavioral data of the AI ​​agent in real time and extract its current cognitive features; and determine whether there is cognitive tampering by comparing the current cognitive features with the dynamic evolution baseline in both overall and local dimensions. The tampering location module is configured to: when cognitive tampering is detected, accurately locate the tampered cognitive dimension and the degree of tampering; The security recovery module is configured to restore the AI ​​agent's self-cognition to a normal state using a tiered recovery strategy based on the degree of tampering.

8. The AI ​​agent self-awareness tampering monitoring and recovery system as described in claim 7, characterized in that, The system also includes an optimization iteration module, which continuously optimizes the preset parameters in the cognitive baseline modeling module, metacognitive consistency monitoring module, and security recovery module, as well as the attack behavior feature library, based on log data generated during the monitoring, location, and recovery process.

9. A computer-readable storage medium, characterized in that, Used to store computer instructions, which, when executed by a processor, perform the method described in any one of claims 1-6.

10. A computer program product, characterized in that, Includes a computer program, which, when executed by a processor, implements the method described in any one of claims 1-6.

Images