Proxy-based endpoint trust monitoring and data protection method and system for large model compatible interface

Abstract

The application relates to a proxy type endpoint trusted monitoring and data protection method and system facing a large model compatible interface, and the method comprises the following steps: identifying, by a proxy service layer, to-be-protected data in an original request, replacing the to-be-protected data with a placeholder mark to obtain a desensitization request, and saving a replacement mapping relationship table; sending the desensitization request to a target upstream endpoint; receiving, by the proxy service layer, an original response returned by the target upstream endpoint, performing stream response processing or non-stream response processing, and obtaining a response output text; replacing, by the proxy service layer, the placeholder mark in the response output text with the to-be-protected data to obtain placeholder recovery response content, and forwarding the placeholder recovery response content to a client; and periodically performing, by a drift detection module of the proxy service layer, target upstream endpoint behavior consistency drift detection in each request-response interaction process. The application realizes effective coordination and balance among data protection, upstream processing availability and response result recoverability.

Description

Technical Field

[0001] This invention relates to the field of large model technology, and in particular to a proxy endpoint trusted monitoring and data protection method and system for large model compatible interfaces. Background Technology

[0002] With the rapid development of large-scale model applications, more and more applications are no longer directly calling the native interfaces of a single model service provider, but instead accessing different model services, cloud platform services, or third-party relay services through compatible interfaces. Currently, OpenAI-style interfaces have become one of the de facto universal access methods in the industry, with common calling methods including Chat Completions and Responses interfaces. OpenAI's official documentation publishes relevant interface specifications and explicitly supports accessing model services through a unified SDK (Software Development Kit) and interface forms; Amazon Bedrock also provides OpenAI-compatible APIs for migrating existing OpenAI-accessed applications with minimal modifications; Azure OpenAI also provides Responses APIs and related REST interfaces (representational state transition interfaces) that are integrated with the OpenAI system. Therefore, unified access, forwarding, and adaptation based on "large-scale model compatible interfaces" have become an important engineering approach for current model service calls.

[0003] Against this technological backdrop, numerous proxy, gateway, and relay solutions have emerged around large model compatibility interfaces. These solutions typically reside between the client and the target model service endpoint, shielding underlying differences through a unified interface format to achieve capabilities such as multi-model switching, unified authentication, routing, fallback, cost control, and access management. For example, LiteLLM (Lightweight Large Model) describes its Proxy Server as a self-hosted OpenAI-compatible gateway, enabling access to multiple large model providers without modifying client-side code; OpenRouter also provides a single unified interface, allowing developers to access multiple models via the OpenAI SDK; Langfuse documentation similarly indicates that many external LLM (Large Language Model) services and proxies can be accessed via the OpenAI API. These findings demonstrate that proxy-based access based on compatible interfaces is already relatively mature in current technologies.

[0004] However, existing proxy and gateway solutions primarily address the issues of "how to unify access" and "how to forward requests." In actual production environments, two other increasingly prominent technical requirements exist. The first is data protection. When actually calling large models, the prompts, context messages, business fields, and attachment metadata submitted by the client often contain sensitive data such as identity information, contact information, access tokens, database connection information, and business-specific content. If the proxy or relay only performs transparent forwarding, the original content will be directly exposed to the upstream service endpoint. To address this, the industry has developed solutions for PII (Personally Identifiable Information) identification, data masking, and sensitive content interception for LLM gateways or call chains. For example, Helicone provides LLM Security capabilities, Gravitee has disclosed a PII Filtering Policy for AI Prompts, and HuggingFace has also disclosed examples of detecting and cleansing personally identifiable information in an LLM Gateway. This indicates that preprocessing and protecting request content in large-scale proxy chains has become an important focus of existing technologies.

[0005] The second type of requirement is endpoint trustworthiness and stability. In compatible interfaces and third-party relay environments, clients typically only see the declared model name and interface format, making it difficult to confirm whether the target upstream endpoint has undergone model replacement, routing changes, inference engine changes, quantization method changes, behavioral drift, or output capability degradation during operation. Traditional API (interface) operation and maintenance monitoring mostly focuses on availability, latency, throughput, and error rate, but these metrics cannot reflect whether model behavior remains consistent. Existing research indicates that even if an endpoint remains "healthy" in terms of uptime, latency, and throughput, its effective model identity may still change due to variations in weights, tokenizers, quantization, inference engines, caching, routing, or hardware. This research proposes a black-box stability monitoring approach based on a fixed set of hints and comparison of output distribution. This study demonstrates that the behavioral stability and identity consistency of large model endpoints have become a new technical problem distinct from traditional interface monitoring. Furthermore, several research works have emerged regarding the identification and fingerprinting of large models, such as the identification and characterization of LLMs using a black-box approach; and the proposal to use under-trained tokens for model fingerprinting. Recently, research has also appeared on different approaches, including behavioral fingerprinting and implicit fingerprinting of LLMs. From an academic research perspective, model fingerprinting and model identity verification have gradually become key research hotspots in the current technological field.

[0006] Existing proxy or relay solutions for large model-compatible interfaces typically focus on request forwarding, interface adaptation, and unified access. They lack a continuous monitoring mechanism for the trustworthiness of the target upstream endpoint, making it difficult to determine if the upstream endpoint is experiencing model replacement, service capability changes, output behavior drift, or abnormal responses during operation. Meanwhile, client requests to large model services often contain sensitive data such as identity information, contact details, access keys, and business data. Existing solutions usually send the original request directly to the upstream endpoint or perform only simple, irreversible data anonymization, failing to simultaneously address data protection, upstream processing availability, and response recoverability. Furthermore, existing technologies largely operate independently of data protection, service monitoring, and model validation solutions, lacking a unified mechanism for collaborative processing within the same proxy chain. This hinders the formation of a closed-loop control process from request entry, proxy forwarding, response return, to continuous monitoring and anomaly handling throughout the runtime. In this situation, while clients can access different large model services through compatible interfaces, the security of request data remains inadequate, and continuous and accurate assessment of the target upstream endpoint's actual service status and trustworthiness is impossible. Summary of the Invention

[0007] To address at least some of the aforementioned problems in the prior art, this invention provides a proxy endpoint trusted monitoring and data protection method for large model compatible interfaces, comprising: The proxy service layer receives the original request sent by the client according to the large model compatible interface format, and determines the corresponding target upstream endpoint from multiple upstream endpoints; The proxy service layer identifies the data to be protected in the original request, replaces the data to be protected with placeholders, obtains the de-identification request, and saves the replacement mapping table; The proxy service layer constructs the HTTP request header and combines it with the de-identified request to form the final upstream request, which is then sent to the target upstream endpoint. The proxy service layer receives the raw response returned by the target upstream endpoint, performs streaming or non-streaming response processing, and obtains the response output text. The proxy service layer replaces the placeholders in the response output text with the data to be protected based on the replacement mapping table, obtaining the placeholder recovery response content, and forwards it to the client, completing one request-response interaction; and During each request-response interaction, the drift detection module of the proxy service layer periodically performs drift detection on the behavior consistency of the target upstream endpoint, including: sending a preset fixed probe prompt to the target upstream endpoint, collecting probe responses, calculating the text similarity and divergence between the current response and the historical output text of the same probe, and making a drift classification judgment based on the divergence calculation results.

[0008] Furthermore, the data to be protected in the original request is identified, and the data to be protected is replaced with placeholders to obtain the de-identified request, which includes: Iterate through all text fields in the original request; perform multi-type sensitive entity rule matching on each text field to filter out sensitive entities; generate a unique placeholder for each matched sensitive entity, wherein generating the placeholder includes: concatenating the type identifier of the current matching rule, the counter value of the current replacement operation, and a randomly generated salt value, calculating a 256-bit digest using a secure hash algorithm, taking the first few characters of the digest, and combining them with a fixed prefix and counter number to form the placeholder; The replacement mapping table is stored in the local data storage module. Each record in the replacement mapping table contains: placeholder, original sensitive entity, matching rule type, and position information in the original request. The de-identification operation will be recorded as a risk event in the risk event table of the local data storage module. The proxy service layer replaces the placeholders in the response output text with the data to be protected based on the replacement mapping table, resulting in a placeholder recovery response that includes: The proxy service layer scans the response output text and identifies placeholders in the response output text through full format matching or naked token compatibility matching. The proxy service layer restores the placeholder markers to the corresponding sensitive entities based on the replacement mapping table.

[0009] Furthermore, the drift detection module of the proxy service layer periodically performs consistency drift detection on the upstream endpoint behavior of the target, including: The drift detection module has a built-in probe-based question bank; The timer triggers the proxy service layer to read the policy configuration, determine whether the drift detection module is started, and if so, it iterates through all enabled target upstream endpoints, performing probe request sending and response collection for each target upstream endpoint, including: Select one or more probe hints from the probe hint question bank and send a probe request to the target upstream endpoint in the standard format of the large model compatible interface; The proxy service layer receives the response content returned by the target upstream endpoint for each probe request and extracts the text portion of the response, i.e., the probe response text; The proxy service layer preprocesses the probe response text to obtain normalized probe response output text: converting all text to lowercase; removing punctuation marks from the text; removing code block markers from the text; and compressing consecutive whitespace characters into a single space. The text similarity between the current normalized probe response output text and the historical output text of the same probe is calculated, and the divergence is calculated based on the text similarity calculation results. Drift classification is determined based on the divergence calculation results, including: If the calculated divergence is less than the first threshold, it is considered to be in a normal state. When the divergence calculation result is greater than or equal to the first threshold and less than the second threshold, it is judged as a sign of drift; When the divergence calculation result is greater than or equal to the second threshold and less than the third threshold, it is determined that it has drifted; When the divergence calculation result is greater than or equal to the third threshold, it is judged as high risk.

[0010] Furthermore, each probe prompt in the probe question bank also includes format expectation information. When the proxy service layer performs format expectation verification after each probe request sending and response collection, it checks whether the response output of the target upstream endpoint meets the format expectation of the probe prompt, and records the format expectation verification result as auxiliary metadata in the probe running record.

[0011] Furthermore, it also includes: when the drift detection module detects that the behavior consistency status of the target upstream endpoint meets the preset triggering conditions, or when the administrator manually triggers the audit, the proxy service layer starts the model fingerprint audit module to perform model identity verification on the target upstream endpoint.

[0012] Furthermore, the preset triggering condition is met when the drift classification determination reaches the drift level or the high-risk level; The model identity verification is performed by the model fingerprint auditing module using the B3IT boundary input black-box identification test method, including: Identify the target upstream endpoints and their declarative models for the audit, and find the corresponding reference distribution baselines; Load the set of boundary input prompts in the reference distribution baseline, and collect the first token or first word output by repeatedly sending multiple requests to the target upstream endpoint for each prompt using deterministic parameters, forming the empirical output distribution of the target upstream endpoint on each prompt; For each boundary input prompt, calculate the total variation distance between the empirical distribution of the upstream endpoint of the target and the baseline reference distribution, and calculate the p-value through a permutation test to assess the statistical significance of the difference; By combining the TVD and p-values ​​from all boundary input prompts, an audit conclusion is generated, and a confidence score is given. The audit conclusion includes: model matching, model mismatch, or uncertainty. The sources of the reference distribution baseline include: establishing it through online collection, including: the administrator selecting an endpoint of a known model, the proxy service layer sampling the endpoint multiple times using all boundary input prompts, statistically analyzing the first token output distribution of each boundary input prompt, and saving it to the fingerprint baseline table in the local database; or The proxy service layer has a built-in set of pre-collected large model reference distribution information as a reference distribution baseline.

[0013] Furthermore, determining the corresponding target upstream endpoint from multiple upstream endpoints includes: After parsing the model name field in the original request, the proxy service layer calls the model name matching routing method in the endpoint storage module to find the upstream endpoint that best matches the model name in the current request among all configured upstream endpoints as the target upstream endpoint. The model name matching routing method uses a multi-level matching scoring mechanism, with the following priorities from highest to lowest: First priority, exact match: compare the model name in the request with the model name declared in the configuration of each upstream endpoint. If they are exactly the same, the model name is directly used as the target upstream endpoint. Second priority, model name suffix matching: Determine whether the model name in the original request is the suffix of the endpoint declaration model name. If so, it is considered a suffix match. The third priority is substring inclusion matching: determining whether the model name in the original request is contained within the endpoint declaration model name; Fourth priority, despeccable matching: compare the model name in the original request and the endpoint declaration model name after removing the separator; Fifth priority, token hit matching: split the model name and endpoint declaration model name in the original request into multiple tokens by the separator, calculate the hit rate of the common tokens between the two model names, and take the one with the highest hit rate.

[0014] Furthermore, before replacing the placeholder data to be protected, the process includes: performing a session tag interception check on the original request; if any session tag carried in the current request matches a blacklist entry, an error response is returned to the client; and The original request is stripped of its local content, and the local content fragments are replaced with placeholder prompts. Before forwarding the placeholder recovery response content to the client, the following is also included: The proxy service layer performs high-risk content detection on the placeholder recovery response content. If the detection passes, the placeholder recovery response content is directly returned to the client. If high-risk content is detected, the risk event is recorded in the local database, and the handling method is determined according to the blocking policy parameters in the policy configuration, and corresponding processing is performed, including: If the blocking policy parameter is set to mark but not block, the proxy service layer will still return the placeholder recovery response content to the client after logging the risk event; If the blocking policy parameter is set to block, the proxy service layer will not forward the original response of the target upstream endpoint to the client, but will construct a blocking response locally in the proxy service layer and return it to the client with an HTTP 451 status code. The built-in rule engine performs high-risk content detection on the placeholder recovery response content. The built-in rule engine contains a set of predefined high-risk inducement pattern detection rules to identify whether there is content in the model output that induces users to disclose sensitive information or perform dangerous operations.

[0015] Furthermore, the proxy service layer receives the raw response returned by the target upstream endpoint, performs streaming response processing, and obtains the response output text, which includes: When the client declares a streaming response in the original request, and the response content type returned by the target upstream endpoint is a streaming response type, the proxy service layer enters the streaming response processing path, including: Initialize streaming statistics variables, including: first token timestamp, streaming event counter, payload character accumulator, output text accumulation buffer, tool call counter, usage pattern record variable, and termination reason record variable; The proxy service layer reads events from the streaming responses returned by the upstream endpoints one by one. For each event, the proxy service layer performs two operations simultaneously: it transmits the event data to the client's response connection as is, ensuring the client receives the streaming response output in real time; and it parses the event payload locally, including: the proxy service layer performs the following analysis on the payload of each event: if the event is the first event containing valid text content, it records the time difference between the current time and the request sending time as the first token time; it extracts the incremental text content fragments from the event payload and appends them to the output text accumulation buffer; it detects and accumulates tool call-related fragments in the event payload; it accumulates the event count and payload character count to the corresponding counters; if the event payload contains a usage field, it records the usage morphology characteristics; if the event payload contains an end reason field, it records the end reason value. After the streaming response output content has been transmitted, the proxy service layer performs placeholder restoration processing on the complete output text spliced ​​in the output text accumulation buffer.

[0016] Furthermore, the proxy service layer receives the raw response returned by the target upstream endpoint, performs non-streaming response processing, and obtains the response output text: When the client's original request does not declare streaming or the response returned by the target upstream endpoint is in standard JSON format, the proxy service layer enters the non-streaming response processing path, including: The proxy service layer reads the entire response returned by the target upstream endpoint as text, thus obtaining the response text; The received response text is parsed into JSON to obtain the response content; Furthermore, it also includes: for each request-response interaction, the proxy service layer extracts large model behavior feature data and writes it into the request log table of the local database; the feature fields in the request log table include the first token time, output text length, number of tokens consumed in the request and the number of tokens generated in the response, number of tokens generated per second, termination reason, JSON format validity, number of tool calls, rejection detection flag, rejection template hash value, usage pattern fingerprint, error response fingerprint, and tool name fingerprint; Before replacing the placeholder data to be protected, the following steps are also included: The original request is intercepted and checked for session tags. If any session tag carried in the current request matches a blacklist item, an error response is returned to the client. The original request is stripped of its local content, and the local content fragments are replaced with placeholder prompts.

[0017] This invention also provides a proxy endpoint trusted monitoring and data protection system for large model compatible interfaces, deployed between the client and multiple upstream endpoints, including: The transparent protocol proxy module is configured to perform the following actions: receive the original request sent by the client in a large model-compatible interface format, parse the model name field in the original request to determine the corresponding target upstream endpoint; and route the de-identified request processed by the local data protection module to the corresponding target upstream endpoint. The local data protection module is configured to perform the following actions: before the original request is sent to the target upstream endpoint, perform local-specific content stripping, session tag interception and inspection, and decoy tag injection operations on the original request; identify the data to be protected in the original request, replace the data to be protected with placeholder tags, generate a de-identified request, and store the replacement mapping relationship between the placeholder tags and the data to be protected in the local data storage module; and receive the response result returned by the target upstream endpoint, and replace the placeholder tags in the response result with the original data to be protected according to the replacement mapping relationship. The drift detection module is configured to perform the following actions: during each request-response interaction, periodically and actively probe the upstream endpoint of the target with a fixed probe cue set, calculate the text similarity and divergence between the current probe output and the historical output text of the same probe, and make a drift classification judgment based on the divergence calculation results, thereby identifying whether there is behavioral drift at the upstream endpoint of the target.

[0018] The model fingerprint audit module is configured to perform the following actions: when the drift detection module detects behavioral drift at the target upstream endpoint or when the administrator manually triggers the audit, it performs model identity verification on the target upstream endpoint.

[0019] Furthermore, it also includes a behavior feature extraction module, which is configured to perform the following actions: extract structured large model behavior feature data from the response returned by the target upstream endpoint and write it to the request log table of the local data storage module.

[0020] The present invention has at least the following beneficial effects: This invention employs a reversible placeholder replacement mechanism to replace sensitive entities with opaque placeholders before the request leaves the server, ensuring that the original sensitive data never leaves the local proxy service layer. Simultaneously, because the placeholders retain the semantic location information "an entity exists here" within the request text, the large model of the target upstream endpoint can still complete the processing task based on the semantic skeleton. After the response is returned, the system reverse-engineers the placeholders back to their original content based on the locally stored replacement mapping relationship, allowing the client to ultimately obtain a usable response containing complete original information. Therefore, this invention achieves an effective balance between data protection, upstream processing availability, and response recoverability, rather than forcing a trade-off between protection strength and availability as is common in existing technologies.

[0021] The method and system of this invention enable continuous and proactive monitoring of the consistency of upstream endpoint behavior, filling the gap in existing proxy gateway solutions for endpoint trust status identification. This invention continuously runs proactive probe detection tasks through a drift detection module, periodically sending probe requests to the upstream endpoint using a fixed probe question bank covering multiple model capability boundary dimensions. It compares the current probe output with historical outputs of the same probe based on Dice coefficients, calculates the divergence, and performs multi-level threshold determination. This method does not rely on external embedded models or additional large models, performing lightweight monitoring only locally at the proxy service layer. Furthermore, it employs a design that separates format expectation verification and drift score to reduce the false positive rate. Therefore, this invention enables the proxy layer to possess continuous, proactive, and lightweight monitoring capabilities for the behavior status of upstream endpoints, allowing it to detect substantial changes in endpoint behavior even when traditional operational indicators show "healthy."

[0022] The method and system of this invention implement a layered triggering mechanism of "lightweight monitoring first, then in-depth verification for anomalies," improving the accuracy of endpoint trustworthiness assessment and resource utilization efficiency. This invention employs a layered triggering design: under normal conditions, only lightweight drift detection is run; only when the drift level reaches a preset trigger condition is a B3IT-based model fingerprint audit initiated to perform model identity verification on the target upstream endpoint. This layered mechanism allows the system to maintain low resource consumption during the majority of endpoint behavior cycles, allocating in-depth verification resources only when necessary, thus achieving a balance between the accuracy of trustworthiness assessment and system operating efficiency.

[0023] This invention integrates data protection, proxy forwarding, response recovery, behavior monitoring, identity verification, and risk handling into a single proxy link, overcoming the problem of existing technologies where each capability is independent and difficult to form a closed loop. All the above-mentioned links form a complete closed-loop control process around the same upstream endpoint, and risk events are uniformly aggregated into a single data table for correlation analysis and source tracing. Compared to the single-point governance methods of existing technologies, the integrated closed-loop architecture of this invention enables information transmission and collaborative triggering between links, resulting in an overall governance effect superior to the simple summation of individual parts.

[0024] This invention can achieve all governance functions while keeping the client calling method completely unchanged, thus reducing the transformation costs of deployment and access.

[0025] This invention uses a built-in rule engine to perform high-risk content detection on placeholder recovery responses along the response path. It can identify content in the model output at the proxy layer that might induce users to disclose sensitive information or perform dangerous operations, and proactively construct a blocking response locally at the proxy service layer to return to the client based on policy configuration, instead of passing through the high-risk content. This means that the proxy layer has the ability to proactively intervene and manage the response path, intercepting risky content before it reaches the client, thus elevating the approach from passive "post-event detection" to proactive "in-event blocking."

[0026] This invention extracts additional structured features from each request-response interaction, focusing on the behavior of the large model. These features include first token time, rejection template hash value, tool name fingerprint, usage pattern fingerprint, error response fingerprint, JSON format validity, termination reason, and streaming event statistics. These feature fields enable the system to build endpoint profiles from the perspective of model output behavior, rather than simply recording access information at the network and interface levels. For example, changes in the rejection template hash value can serve as an auxiliary signal that the model may be replaced; changes in the usage pattern fingerprint can indicate adjustments to upstream service implementations; and changes in the tool name fingerprint can reflect differences in the model's tool invocation capabilities. This rich behavioral feature data not only provides more dimensional signal inputs for drift detection and trustworthiness determination but also provides administrators with a data foundation far exceeding existing solutions for long-term operational trend analysis and anomaly tracing.

[0027] This invention unifies the following capabilities, originally belonging to different technical solutions, into a single proxy processing chain oriented towards a large model-compatible interface: session tag interception, local exclusive content stripping, identification and placeholder replacement of data to be protected, proxy forwarding, streaming and non-streaming response processing and recovery, high-risk content detection and proactive blocking on the response side, behavioral feature extraction and structured log recording, periodic behavioral consistency drift detection, model identity verification under abnormal triggering, and comprehensive trust determination and risk handling. These links form a complete closed loop around the same upstream endpoint, from request entry to continuous monitoring and abnormal handling throughout the runtime cycle. Clients only need to point the interface base address to the proxy service to access it, without changing their original calling method. Attached Figure Description

[0028] To further illustrate the above and other advantages and features of the various embodiments of the present invention, a more specific description of the embodiments of the invention will be presented with reference to the accompanying drawings. It is to be understood that these drawings depict only typical embodiments of the invention and are therefore not intended to limit its scope. In the drawings, identical or corresponding parts will be indicated by identical or similar reference numerals for clarity.

[0029] Figure 1 A schematic diagram of a proxy endpoint trusted monitoring and data protection system for large model compatible interfaces is shown.

[0030] Figure 2A This paper illustrates the complete processing flow from receiving a client request to returning a response in a proxy endpoint trusted monitoring and data protection method for a large model compatible interface according to an embodiment of the present invention.

[0031] Figure 2B The diagram illustrates the background periodic task flow for running the monitoring dimension in a proxy endpoint trusted monitoring and data protection method for a large model compatible interface according to an embodiment of the present invention.

[0032] Figure 3 A flowchart illustrating proxy access and request routing according to an embodiment of the present invention is shown.

[0033] Figure 4 A flowchart illustrating the identification and placeholder replacement of data to be protected according to an embodiment of the present invention is shown.

[0034] Figure 5 A flowchart of a streaming response processing method according to an embodiment of the present invention is shown.

[0035] Figure 6 A flowchart of response recovery and high-risk detection according to an embodiment of the present invention is shown.

[0036] Figure 7 A flowchart of behavior consistency drift detection according to an embodiment of the present invention is shown. Detailed Implementation

[0037] It should be noted that the components in the accompanying drawings may be shown exaggerated for illustrative purposes and may not be to scale.

[0038] In this invention, the various embodiments are merely intended to illustrate the solutions of the invention and should not be construed as limiting.

[0039] In this invention, unless otherwise specified, the quantifiers “a” and “one” do not exclude scenarios involving multiple elements.

[0040] It should also be noted that, in the embodiments of the present invention, only a portion of the parts or components may be shown for clarity and simplicity. However, those skilled in the art will understand that, under the teachings of the present invention, the required parts or components can be added as needed for specific scenarios.

[0041] It should also be noted that within the scope of this invention, the terms "same", "equal", and "equal to" do not mean that the two values ​​are absolutely equal, but allow for a certain reasonable error. In other words, the terms also cover "substantially the same", "substantially equal", and "substantially equal to".

[0042] It should also be noted that in the description of this invention, the terms "center," "longitudinal," "lateral," "upper," "lower," "front," "rear," "left," "right," "vertical," "horizontal," "top," "bottom," "inner," and "outer," etc., indicate the orientation or positional relationship based on the orientation or positional relationship shown in the accompanying drawings. They are used only for the convenience of describing the invention and for simplifying the description, and do not explicitly or implicitly suggest that the device or element referred to must have a specific orientation, or be constructed and operated in a specific orientation. Therefore, they should not be construed as limitations on the invention. Furthermore, the terms "first" and "second" are used for descriptive purposes only and should not be construed as indicating or implying relative importance.

[0043] Furthermore, the embodiments of the present invention describe the process steps in a specific order. However, this is only for the convenience of distinguishing each step, and is not a limitation on the order of each step. In different embodiments of the present invention, the order of each step can be adjusted according to the process.

[0044] The existing technical solutions related to this invention can be mainly categorized into the following three types.

[0045] The first type of existing solution: Large model-compatible interface proxy or gateway solution. This type of solution typically sets up a unified proxy layer or gateway layer between the client and multiple model service providers. Externally, it exposes an OpenAI-style interface, while internally handling model routing, protocol adaptation, authentication management, access control, logging, and fault rollback. Its basic implementation is generally as follows: the client still initiates a call according to the OpenAI SDK or a compatible format. The request first reaches the proxy service, which selects the target model service or upstream endpoint according to its preset configuration, forwards the request to the corresponding target, and finally returns the result to the client in a compatible format. LiteLLM and OpenRouter belong to this category. The advantage of this type of solution is that it keeps the client's calling interface basically unchanged and reduces the complexity of multi-model, multi-platform access. However, its focus is usually on interface unification, routing, and access management, rather than "trustworthy monitoring of the target upstream endpoint" as its core objective. The client can generally only see the declared model name, call results, and routine operating metrics, making it difficult to determine whether the target endpoint has undergone actual model changes or output behavior drift during the runtime.

[0046] The second category of existing solutions focuses on data security and observation of the LLM call chain. These solutions typically address security or observability by adding capabilities such as request logging, response logging, prompt management, sensitive information detection, PII masking, content interception, and anomaly alerts to the call chain. For example, Helicone's documentation discloses an observation scheme for recording and analyzing LLM requests through a proxy, and also provides LLM security capabilities; Gravitee has disclosed strategies for filtering personally identifiable information in AI prompts; some public practices also demonstrate scrubbing (de-identification) or sanitizing data at the gateway before sending it to the model endpoint. The basic idea is to detect, filter, or replace the input content before the request reaches the model service, and to record or intercept it during the response phase using relevant strategies. This type of solution helps reduce the risk of sensitive information in prompts being directly exposed to the model service and also enhances the logging and governance capabilities of the call chain. However, most existing solutions lean more towards "security filtering" or "observation logging," typically without further attention to whether the target upstream endpoint is consistent with the declared model, nor do they build a mechanism to monitor endpoint behavior consistency throughout the runtime. In other words, this type of solution mainly addresses the issues of "whether the content sent out is secure" and "whether the request is observable," but does not address the issue of "whether the upstream endpoint itself is trustworthy."

[0047] The third category of existing solutions includes model fingerprinting, endpoint stability, and identity analysis. These solutions primarily analyze the identity, origin, and behavioral stability of model ontologies or service endpoints from a research or testing perspective. The implementation involves inputting a fixed set of prompts into the model under test, collecting output results, output distribution, or behavioral characteristics; then comparing the current results with pre-established reference features, baseline model features, or historical behavioral characteristics to determine model affiliation, model differences, the likelihood of model replacement, or whether endpoint behavior has changed. For example, some existing studies propose periodic fingerprinting of endpoints and using output distribution differences as evidence of stability; others discuss LLM fingerprinting under black-box conditions; still others propose using under-trained token features for model identification. While these solutions can identify model differences or discover endpoint behavioral changes from an academic or testing perspective, most existing solutions are not deployed in compatible interface proxy links for actual business calls and are typically not designed in conjunction with sensitive data protection, response recovery, and unified risk handling on the request side. It focuses more on "how to detect whether the model or endpoint has changed" rather than "how to integrate data protection, endpoint monitoring, identity verification and risk governance into the same proxy process without changing the way the client makes calls".

[0048] In summary, while existing technologies can achieve unified access, request forwarding, data filtering, and runtime observation for large model-compatible interfaces to a certain extent, they still have the following shortcomings: (1) Existing compatible interface proxy solutions lack the ability to continuously identify the trusted status of the target upstream endpoint. Existing large-model compatible interface proxy or gateway solutions mainly focus on request forwarding, protocol adaptation, route management and unified access. Their core goal is to improve the convenience and engineering compatibility of multi-model service access. Such solutions usually assume that the model capabilities declared by the target upstream endpoint are consistent with the actual service capabilities provided, and lack a continuous monitoring mechanism for whether the target upstream endpoint undergoes model replacement, service capability changes, behavior drift or output anomalies during operation. Therefore, under the current technical conditions, the client can usually only know whether the request was successfully forwarded and whether the interface is available, but it is difficult to further determine the actual trusted status of the target upstream endpoint.

[0049] (2) Existing data security solutions struggle to balance the security of the data to be protected, the availability of upstream processing, and the recoverability of response results. Existing data security solutions for large-scale model call chains typically employ methods such as direct interception, static masking, simple replacement, or irreversible desensitization. While these methods can reduce the risk of direct exposure of sensitive information to some extent, they often present two problems: first, the processed request content may lose its original semantic structure or key context, affecting the processing effect of the target upstream endpoint; second, it is difficult to effectively recover the response based on the processing results of the request stage, thus affecting the client's actual use of the response results. Therefore, existing technologies still suffer from insufficient coordination between data protection and result availability.

[0050] (3) Existing endpoint monitoring, data protection, and model verification technologies are independent of each other and lack a unified coordination mechanism. In existing technologies, capabilities such as compatible interface proxy, sensitive information processing, call observation, model behavior analysis, and anomaly alerting are usually implemented by different solutions, lacking a unified processing link and coordinated triggering mechanism. For example, data protection usually only applies before the request is sent, operation monitoring usually only focuses on call logs or service status, and model identity analysis usually exists as an independent detection method, making it difficult to form a closed-loop control process from request entry, data processing, proxy forwarding, response recovery to anomaly triggering and risk handling in the same proxy call process. As a result, existing technologies are still biased towards single-point governance and cannot meet the comprehensive requirements of data security, endpoint trustworthiness, and operational controllability in large model compatible interface scenarios.

[0051] (4) Existing technologies struggle to achieve integrated trusted governance while maintaining the client's calling method unchanged. While some existing solutions can address specific aspects of data protection, access control, or anomaly detection, they typically require additional modifications to the client's calling method, interface parameters, deployment mode, or business processes, or they only work in specific scenarios, lacking versatility and compatibility. For clients already using large-model compatible interfaces for access, there is a lack of an integrated solution that can simultaneously achieve data processing to be protected, trusted monitoring of the target upstream endpoint, anomaly verification, and risk handling without altering their original calling method.

[0052] To address the aforementioned shortcomings, the present invention aims to provide a proxy endpoint trusted monitoring and data protection method and system for large model compatible interfaces. This system achieves the following objectives while maintaining the basic client-side compatible interface calling method: securely processing the data to be protected in the request to reduce the risk of direct exposure of the original data to the target upstream endpoint; enabling recoverable processing of response results without significantly affecting upstream processing performance, thereby improving the availability of proxy call results; continuously monitoring the target upstream endpoint throughout its runtime cycle to identify changes in its behavioral state; further verifying the model identity or service capabilities of the target upstream endpoint when an abnormal state is detected, thereby improving the ability to judge endpoint trustworthiness; and unifying data protection, proxy forwarding, state monitoring, anomaly verification, and risk handling into a single proxy link, forming an integrated closed-loop governance mechanism for large model compatible interface scenarios.

[0053] Figure 1 A schematic diagram of a proxy endpoint trusted monitoring and data protection system for large model compatible interfaces is shown.

[0054] like Figure 1 As shown, the proxy endpoint trusted monitoring and data protection system for large model compatible interfaces is deployed between client 10 and multiple upstream endpoints, belonging to the proxy service layer 20. The client communicates with the proxy service layer through the large model compatible interface, and the proxy service layer forwards requests to the corresponding target upstream endpoints according to the routing results.

[0055] The proxy service layer 20 provides external interface addresses (access addresses) that conform to the large model compatibility interface specification. The interfaces exposed by the proxy service layer 20 include, but are not limited to, the chat completion interface (Chat Completions, path / v1 / chat / completions), the response generation interface (Responses, path / v1 / responses), and the model list query interface (path / v1 / models).

[0056] Client 10 only needs to adjust the base address (base_url) of the calling interface to point to the interface address of the proxy service layer to quickly access the system of the present invention without changing the original calling code and calling method.

[0057] The proxy endpoint trusted monitoring and data protection system for large model compatible interfaces includes a transparent protocol proxy module 201, a local data protection module 202, a behavior feature extraction module 203, a drift detection module 204, a model fingerprint audit module 205, and a local data storage module 206.

[0058] The transparent protocol proxy module 201 is configured to perform the following actions: receive the original request sent by the client in a large model-compatible interface format, and parse the model name field in the original request to determine the corresponding target upstream endpoint; Based on the endpoint configuration information, the de-identification request processed by the local data protection module 202 is routed to the corresponding target upstream endpoint.

[0059] Local data protection module 202 is configured to perform the following actions: Before the original request is sent to the target upstream endpoint, local exclusive content stripping, session tag interception and inspection, and decoy tag injection operations are performed on the original request; Identify the data to be protected in the original request, replace the data to be protected with placeholders, generate a de-identification request, and store the replacement mapping relationship between the placeholders and the data to be protected in the local data storage module; Receive the response result returned by the upstream endpoint of the target, and replace the placeholders in the response result with the original data to be protected according to the replacement mapping relationship; The behavior feature extraction module 203 is configured to perform the following actions: Extract structured large model behavioral feature data from the response returned by the target upstream endpoint and write it to the request log table of the local data storage module 206.

[0060] The behavioral feature data of the large model includes, but is not limited to, output text length, JSON (JavaScript Object Notation) format validity, number of tool calls, rejection template hash value, usage shape, error fingerprint, Time To First Token (TTFT), streaming event statistics, etc., forming a structured log for the behavior side of the large model.

[0061] The drift detection module 204 (internal name Sentinel) is configured to perform the following actions: During each request-response interaction, the upstream endpoint of the target is periodically and actively probed with a fixed probe cue set. The text similarity and divergence of the current probe output and the historical output text of the same probe are calculated. Based on the divergence calculation results, the drift classification is determined to identify whether there is behavioral drift of the upstream endpoint of the target.

[0062] Specifically, during the operation cycle, the drift detection module 204 sends a preset probe request to the target upstream endpoint, collects the probe response, and compares the current probe response with the historical reference response to determine the consistency status of the target upstream endpoint's behavior, thereby identifying whether the target upstream endpoint has changes in service capabilities, behavior drift, or abnormal output.

[0063] Model fingerprint audit module 205 is configured to perform the following actions: When the drift detection module detects behavioral drift at the target upstream endpoint or when an administrator manually triggers an audit, model identity verification is performed on the target upstream endpoint. Specifically, the target upstream endpoint is sampled multiple times using border input prompts to obtain the model's first token output distribution, which is then statistically compared with a reference distribution baseline. Statistical significance is determined through Total Variation Distance (TVD) calculation and permutation test, ultimately outputting a probabilistic conclusion of model matching, model mismatch, or uncertainty.

[0064] Specifically, the model fingerprint audit module 205 performs model identity or service capability verification under abnormal triggering. When the behavior consistency status meets the preset triggering conditions, it further performs model identity verification or service capability verification on the target upstream endpoint to determine whether the actual service status of the target upstream endpoint is consistent with the declared status, thereby improving the accuracy of endpoint trustworthiness judgment.

[0065] The local data storage module 206 uses an SQLite relational database and enables Write-Ahead Logging (WAL) mode to persistently store endpoint configurations, request logs, risk events, drift events, probe running records, fingerprint baselines, and fingerprint audit results.

[0066] The transparent protocol proxy module 201, the local data protection module 202, the behavior feature extraction module 203, the drift detection module 204, and the model fingerprint audit module 205 are all connected to the local data storage module 206.

[0067] The proxy-based endpoint trusted monitoring and data protection system of the present invention also includes a management console 207, which is a web-based visual management interface used to display information such as endpoint status, risk events, drift trends, and fingerprint audit reports, as well as to manage endpoint configurations and policy parameters. The management console obtains data by calling the internal management interface provided by the proxy service layer and does not participate in the core algorithm processing itself.

[0068] Figure 2A This paper illustrates the complete processing flow from receiving a client request to returning a response in a proxy endpoint trusted monitoring and data protection method for a large model compatible interface according to an embodiment of the present invention. Figure 2B The diagram illustrates the background periodic task flow for running the monitoring dimension in a proxy endpoint trusted monitoring and data protection method for a large model compatible interface according to an embodiment of the present invention.

[0069] like Figure 2A As shown, the complete processing chain from receiving a client request to returning a response includes the following steps: The proxy service layer receives the original request sent by the client according to the large model compatible interface format, and determines the corresponding target upstream endpoint from multiple upstream endpoints; The original request is intercepted and checked for session tags. If any session tag carried in the current request matches a blacklist item, an error response is returned to the client. The original request is stripped of its local content, and the local content fragments are replaced with placeholder prompts. The proxy service layer performs high-risk content detection on the placeholder recovery response content. If the detection passes, the placeholder recovery response content is returned directly to the client. If high-risk content is detected, the risk event is recorded in the local database, and the processing method is determined according to the blocking policy parameters in the policy configuration, and the corresponding processing is performed. The proxy service layer identifies the data to be protected in the original request, replaces the data to be protected with placeholders, obtains the de-identification request, and saves the replacement mapping table; The proxy service layer rewrites the model field in the de-identification request to the model name declared by the target upstream endpoint, and injects a decoy token into the metadata of the de-identification request; The proxy service layer constructs the HTTP request header and combines it with the de-identified request to form the final upstream request, which is then sent to the target upstream endpoint. The proxy service layer receives the raw response returned by the target upstream endpoint, performs streaming or non-streaming response processing, and obtains the response output text. The proxy service layer replaces the placeholders in the response output text with the data to be protected according to the replacement mapping table, obtains the placeholder recovery response content, and forwards it to the client, completing a request-response interaction; Optionally, before forwarding the placeholder recovery response content to the client, the proxy service layer performs high-risk content detection on the placeholder recovery response content. If the detection passes, the placeholder recovery response content is directly returned to the client. If high-risk content is detected, the risk event is recorded in the local database, and the processing method is determined according to the blocking policy parameters in the policy configuration, and the corresponding processing is performed. For each request-response interaction, the proxy service layer extracts large model behavioral feature data and writes it to the request log table in the local database.

[0070] like Figure 2B As shown, the background periodic task flow for the running monitoring dimension includes the following steps: During each request-response interaction, the drift detection module of the proxy service layer periodically performs drift detection of the target upstream endpoint behavior consistency, including: sending a preset fixed probe prompt to the target upstream endpoint, collecting probe responses, calculating the text similarity and divergence between the current response and the historical output text of the same probe, and making a drift classification judgment based on the divergence calculation results; When the drift detection module detects that the behavior consistency status of the target upstream endpoint meets the preset triggering conditions or when the administrator manually triggers the audit, the proxy service layer starts the model fingerprint audit module to perform model identity verification on the target upstream endpoint. The results of data processing to be protected, high-risk content detection, behavior consistency drift detection of the target upstream endpoint, and identity verification of the target upstream endpoint execution model are integrated to generate the overall trust status of the target upstream endpoint, and corresponding risk handling strategies are executed accordingly.

[0071] Figure 3 A flowchart illustrating proxy access and request routing according to an embodiment of the present invention is shown.

[0072] like Figure 3 As shown, proxy access and request routing refer to receiving the original request sent by the client according to the large model compatible interface format and determining which target upstream endpoint the request should be forwarded to.

[0073] The proxy service layer exposes multiple routing paths (request paths) that conform to the large model compatible interface specification. These are interface access paths for different purposes. Clients can call different capability interfaces such as chat generation and model query by filling in the corresponding routing path.

[0074] When a client initiates an HTTP (Hypertext Transfer Protocol) request, the proxy service layer first parses the request path in the HTTP request, determines the interface type based on the request path, and then parses the model name field (model field) from the original request.

[0075] After the model name field is parsed, the proxy service layer calls the model name matching routing method in the endpoint storage module to find the upstream endpoint that best matches the currently requested model name among all configured upstream endpoints as the target upstream endpoint.

[0076] The model name matching routing method employs a multi-level matching scoring mechanism, with the following priorities from highest to lowest: First priority, exact match: The model name in the original request is compared with the model name declared in the configuration of each endpoint. If they are exactly equal, the endpoint is matched directly. Second priority, model name suffix matching: Determine whether the model name in the original request is the suffix part of the model name declared by a certain endpoint. If so, it is considered a suffix match. The third priority is substring inclusion matching: determining whether the model name in the request is contained within the model name declared by a certain endpoint; Fourth priority, de-symbol compact matching: compare the model name in the original request and the endpoint declaration model name after removing hyphens, underscores, periods, and other separators; Fifth priority, token hit matching: split the model name and endpoint declaration model name in the original request into multiple tokens by the separator, calculate the hit rate of the common tokens between the two model names, and take the one with the highest hit rate.

[0077] Matching is performed sequentially based on priority. Once a method with the highest priority matches successfully, subsequent lower-priority matching methods will not be executed. After a successful match, the system returns to the configuration item of the corresponding upstream endpoint in the proxy service layer.

[0078] If the above multi-level matching fails to determine the upstream target endpoint, then the process falls back to the configuration item marked as the default endpoint in the proxy service layer.

[0079] Through the aforementioned multi-level matching and scoring mechanism, the proxy service layer is equipped with the ability to perform intelligent logical routing based on model declarations. This enables it to adapt to the actual situation where different clients use different model name variations for invocation, and ensures that subsequent monitoring, statistics, and auditing operations are performed around the standardized declared model names.

[0080] The following section details the request preprocessing and policy interception judgment (step S200).

[0081] After identifying the target upstream endpoint, the proxy service layer enters the request preprocessing phase and performs the following strategic judgments: (1) Session tag interception and inspection. The proxy service layer extracts session tag information from multiple sources of the original request, including: custom fields in the HTTP request header (e.g., x-relayradar-session-tags) and session tag fields (metadata.session_tag or metadata.session_tags) in the metadata object of the request body. The extracted session tags are compared with the list of disallowed relay session tags (disallowRelaySessionTags) in the policy configuration. If any session tag carried by the current request matches a blacklist item in the list, the proxy service layer directly refuses to forward the request and returns an error response to the client. This mechanism enables the proxy service layer to perform strategic interception based on session-level tag attributes before the request enters the forwarding chain.

[0082] (2) Local-Specific Content Stripping. The proxy service layer scans the prompt text in the request content and identifies local-specific content fragments wrapped in specific tags. The local-specific content tags adopt a predefined format (e.g., [[LOCAL_ONLY: content]]). The proxy service layer matches the tag using regular expressions and replaces the local-specific content fragments wrapped in the tag with fixed placeholder prompts (e.g., [LOCAL_RULE_REDACTED]). After this processing, this part of the content is only retained in the local processing chain and will not be sent to the target upstream endpoint with the request. This mechanism realizes the hierarchical management of prompt content, allowing the client to include content for upstream model processing and private annotations or rule content for local use only in the same prompt.

[0083] Figure 4 A flowchart illustrating the identification and placeholder replacement of data to be protected according to an embodiment of the present invention is shown.

[0084] The identification and replacement of data to be protected (step S300) is the core link of the data protection mechanism of the present invention. Its technical goal is to replace the data to be protected with opaque placeholders before the request is sent out (before it is sent to the target upstream endpoint), and at the same time save the replacement mapping relationship between the data to be protected and the placeholders in the local data storage module for recovery use in the response phase.

[0085] Specifically, the step of identifying and replacing the data to be protected includes the following sub-steps: Sub-step S310: Traverse all text fields in the original request. The text fields include prompt text, content fields of each message in the message list, system prompts, and text type fields in the metadata.

[0086] Sub-step S320 performs multi-type sensitive entity rule matching on each text field to filter out sensitive entities. The multi-type sensitive entity rule matching is implemented using regular expressions and pattern matching. Built-in rule types include: email address, ID card number, phone number, API key or access token, JWT (JSON Web Token) token, database connection string, and private key text block (e.g., the content between the header and footer tags of a PEM-formatted private key). In addition, the system (proxy service layer) also supports specifying a manually configured list of fixed sensitive phrases (manualRedactionStrings) through a policy configuration file, and performing matching and replacement on these phrases as well.

[0087] Sub-step S330 generates a unique placeholder for each matched sensitive entity. The placeholder is generated by taking the type identifier of the current matching rule, the counter value of the current replacement operation, and appending a randomly generated salt value. This information is then concatenated to calculate a SHA-256 (Secure Hash Algorithm 256-bit) digest. The first few characters of the digest are combined with a fixed prefix and the counter number to form the final placeholder. For example, the generated placeholder might look like [[RR_A1B2C3_0001]], where RR is the fixed prefix, A1B2C3 is the truncated hash digest, and 0001 is the counter number. Because the placeholder contains a random salt value and hash operations, different sensitive entities, even with identical content, will generate different placeholders, preventing the placeholder itself from becoming a channel for information leakage.

[0088] Sub-step S340 replaces each sensitive entity in the original request with the corresponding placeholder mark to form the desensitized request content, which is the desensitized request.

[0089] Sub-step S350 involves storing a replacement mapping table in the local data storage module. Each record in this table includes: a placeholder marker, the original sensitive entity, the matching rule type, and its position information in the original request. This replacement mapping table is stored only in the local memory or local storage of the proxy service layer and is not sent to the target upstream endpoint with the request.

[0090] Sub-step S360 records the de-identification operation as a risk event in the risk event table of the local data storage module. The recorded content includes the event type (request side sensitive information de-identification), the type of rule involved, the number of replacements, the associated endpoint identifier and request identifier, etc.

[0091] The technical features of the placeholder replacement mechanism are as follows: First, the placeholder mark retains the semantic location information of "an entity exists here". The large model of the target upstream endpoint can still perceive that there is information placeholder at this position, and will not completely lose the understanding of the semantic structure of the request due to the lack of content, thus ensuring the availability of upstream processing while protecting data; Second, the replacement is reversible. The proxy service layer can restore the placeholder mark to the original content according to the replacement mapping table during the response phase; Third, the original sensitive data never leaves the local proxy layer, reducing the risk of data being directly exposed to the target upstream endpoint.

[0092] The following details the supplementary processing of desensitization requests and outbound preparation (step S400).

[0093] After replacing the placeholder data to be protected and before sending the data masking request to the target upstream endpoint, the system (proxy service layer) performs the following supplementary processing: (1) Forced Model Name Rewriting. When the target upstream endpoint specifies a declared model name in its configuration, the proxy service layer will force the model field in the request body (de-identified request) to be rewritten to the declared model name, regardless of the model name filled in by the client in the original request. This step serves two purposes: first, it avoids inconsistencies between the model name passed by the client and the actual configuration of the target upstream endpoint; second, it ensures that subsequent probing, statistics, and auditing operations are all performed around a unified declared model name, thereby improving data consistency.

[0094] (2) Canary token injection. When the canary token function is enabled in the policy configuration, the system injects a locally generated canary token into the metadata of the de-identified request. The canary token is a unique identifier string randomly generated locally by the proxy service layer (e.g., a token in the form of rrc_xxx). The purpose of the injection is that if the canary token subsequently appears in abnormal response content that should not contain the token, in third-party logs, or in other unexpected locations, it can serve as evidence of improper data propagation in the chain, assisting in source tracing analysis. The system also records the canary injection operation as a risk event.

[0095] (3) Upstream request header construction and authentication processing. The proxy service layer constructs an HTTP request header containing the Authorization request header based on the configuration of the target upstream endpoint and sends it to the target upstream endpoint for authentication. The target upstream endpoint receives the HTTP request header, retrieves the credentials from the Authorization header, and performs authentication.

[0096] Regarding authentication, the system supports two modes: The first mode is the authentication passthrough mode. When the authentication passthrough option (passthroughAuth) is enabled in the endpoint configuration, the system forwards the Authorization request header carried in the client's original request to the target upstream endpoint as is. The Authorization request header is a specific field in the HTTP request header that is used for identity authentication and carries tokens and credentials.

[0097] The second mode is the authentication replacement mode. When the authentication pass-through option is disabled, the system (proxy service layer) prioritizes using the API key (apiKey) stored in plaintext in the endpoint configuration. If no plaintext key is configured, it reads the system environment variable value corresponding to the environment variable reference (apiKeyEnv) specified in the endpoint configuration and uses this value to construct the Authorization request header, which is then sent to the target upstream endpoint. The authentication replacement mode allows the client to use any placeholder key or an empty key locally for invocation, thus allowing the proxy service layer to inject the real upstream access credentials, thereby avoiding the exposure of the real key on the client side.

[0098] (4) Compatibility Parameter Detection and Adaptation. Since different upstream endpoints may have protocol differences when implementing large model-compatible interfaces, the proxy service layer incorporates an automatic parameter compatibility detection mechanism. Specifically, regarding the difference in parameter names for the maximum number of output tokens (for the function of "limiting the maximum number of tokens output by AI," different upstream interfaces require different parameter names; some endpoints accept the `max_tokens` parameter, while others only accept the `max_completion_tokens` parameter), the proxy service layer uses the following detection method: When sending a request to the target upstream endpoint for the first time, the `max_tokens` parameter name is used by default; if the error message returned by the target upstream endpoint contains a prompt indicating that `max_completion_tokens` should be used, the proxy service layer caches this detection conclusion locally and automatically uses the `max_completion_tokens` parameter name for all subsequent requests to that endpoint within a certain time window. The compatibility parameter detection mechanism is reused in the proxy forwarding, drift detection, and model fingerprint auditing modules to ensure that requests sent by each module to the target upstream endpoint are compatible with its protocol requirements.

[0099] After completing all the above supplementary processing, the system will combine the de-identified request content with the constructed request header to form the final upstream request, ready to enter the forwarding stage.

[0100] The following details the steps for forwarding the proxy to the target upstream endpoint (step S500).

[0101] The system (proxy service layer) sends the final upstream request to the interface address of the target upstream endpoint through an HTTP client, receives the HTTP response returned by the target upstream endpoint, and determines the processing path type.

[0102] The proxy service layer calls its internal HTTP client to assemble the final upstream request, and this HTTP client sends the final upstream request to the target upstream endpoint.

[0103] After successful forwarding, the proxy service layer receives the HTTP response returned by the target upstream endpoint and determines whether to enter the streaming response processing path or the non-streaming response processing path based on the content type of the response and whether streaming was declared in the original client request (stream parameter).

[0104] If an exception occurs during forwarding, it will be handled accordingly based on the exception type. During forwarding, the proxy service layer sets up a timeout control mechanism, using a timeout abort signal (AbortSignal.timeout) to limit the request execution time.

[0105] If an anomaly occurs during the forwarding process, the proxy service layer will handle it differently based on the type of anomaly: If the cause of the exception is a timeout response from the upstream endpoint, the proxy service layer returns an HTTP 504 status code (gateway timeout) to the client and records the timeout exception in the request log and risk event table; If the cause of the exception is that the upstream endpoint is unreachable or the network connection fails, the proxy service layer returns an HTTP 502 status code (Error Gateway) to the client and also records the corresponding exception.

[0106] The following details the streaming response processing and non-streaming response processing (step S600a).

[0107] Figure 5 A flowchart of a streaming response processing method according to an embodiment of the present invention is shown.

[0108] like Figure 5 As shown, when the client declares stream=true in the original request (the client requests a streaming response) and the response content type returned by the target upstream endpoint is text / event-stream (streaming response SSE type), the proxy service layer enters the streaming response processing path.

[0109] Specifically, streaming response processing includes the following sub-steps: Sub-step S611 initializes streaming statistics variables, including: first token timestamp (used to record the time interval between the request being sent and the first SSE event containing valid content being received), streaming event counter (records the total number of received events), payload character accumulator (records the total number of characters in all event payloads), output text accumulation buffer (used to concatenate all output text fragments), tool call counter, usage pattern recording variable, and termination reason recording variable.

[0110] Sub-step S612 reads the events in the SSE stream (streaming response) returned by the upstream endpoint of the target one by one. For each SSE event, the proxy service layer performs two operations simultaneously: the first operation is to pass the event data to the client's response connection as is, ensuring that the client can receive the streaming response output in real time without disrupting the streaming experience; the second operation is to parse the payload content of the event locally in the proxy service layer.

[0111] Parse event payload: Extract the JSON-formatted event payload from the current SSE event.

[0112] During local parsing, the proxy service layer performs the following analysis on the payload content of each event: If the event is the first event containing valid text content, the time difference between the current time and the request sending time is recorded as the first token time (TTFT); the text content fragment of the delta portion of the event payload is extracted and appended to the output text accumulation buffer, where the delta portion refers to the newly generated small piece of text; tool call related fragments in the event payload are detected and accumulated; the number of events and the number of payload characters are accumulated to the corresponding counters; if the event payload contains a usage field, the usage morphology characteristics are recorded; if the event payload contains a finish_reason field, the finish_reason value is recorded.

[0113] In sub-step S613, after the SSE stream transmission is completed, the system performs placeholder mark restoration processing on the response output text spliced ​​in the output text accumulation buffer (see step S700 for details), and performs high-risk response detection based on the response output text (see the risk detection section in step S700 for details).

[0114] Sub-step S614: Write all statistical results of this streaming request into the structured request log.

[0115] The technical characteristics of streaming response processing lie in the fact that the proxy service layer achieves parallel processing capabilities—simultaneous transparent transmission, parsing, statistics, and feature extraction—without interrupting streaming transmission. This enables the proxy service layer to possess a deep understanding of the streaming transmission protocol content, rather than simply performing byte-level transparent forwarding. This processing method makes the proxy service layer of this invention a large-scale protocol-aware proxy, capable of completing behavioral feature collection and risk detection even in streaming scenarios.

[0116] The following section details the non-flow response processing (step S600b).

[0117] When the client's original request does not declare streaming or the response returned by the target upstream endpoint is in standard JSON format, the system (proxy service layer) enters the non-streaming response processing path. The specific processing procedure is as follows: In sub-step S621, the proxy service layer reads the entire response returned by the target upstream endpoint as text to obtain the response text.

[0118] Sub-step S622 involves attempting safe JSON parsing (safeJsonParse) on the read response text to obtain the response output text. Formatting exceptions are captured during the parsing process to prevent the proxy service layer from crashing due to non-standard JSON content returned from the upstream.

[0119] Sub-step S623 performs placeholder restoration processing and high-risk response detection on the parsed response content (see step S700 for details).

[0120] Sub-step S624: If the response content is not blocked, the recovered response is sent back to the client according to the original HTTP status code returned by the target upstream endpoint.

[0121] Figure 6 A flowchart of response recovery and high-risk detection according to an embodiment of the present invention is shown.

[0122] See Figure 6 The response recovery and risk detection (step S700) performs two core processes on the response content returned by the upstream endpoint of the target: placeholder reverse recovery and high-risk response detection.

[0123] (a) Reverse recovery of placeholder markers (sub-step S710).

[0124] The proxy service layer scans the response output text to identify placeholders. Two identification modes can be used: the first is full format matching, which identifies complete placeholders in the form of [[RR_XXXXXX_NNNN]], where the double brackets are the delimiters; the second is raw token compatibility matching, which means that even if the target upstream endpoint's large model removes the outer double brackets of the placeholder when generating the response (e.g., the model only outputs RR_XXXXXX_NNNN instead of [[RR_XXXXXX_NNNN]]), the proxy service layer can still identify and recover such raw tokens.

[0125] For each identified placeholder or raw token, the proxy service layer queries the locally stored replacement mapping table to obtain its corresponding original sensitive entity, and performs replacement restoration in the response text to obtain the placeholder restoration response content. After restoration, the placeholder restoration response content finally received by the client does not contain the placeholder token.

[0126] The raw token-compatible recovery mechanism is a key technical detail of this invention. In practical applications, when processing inputs containing placeholders, the large model of the target upstream endpoint may reference these placeholders in the output but modify their format (e.g., removing square brackets, changing capitalization, etc.). The raw token-compatible recovery mechanism enables the proxy service layer to correctly complete the recovery process in such cases, improving the robustness of the recovery mechanism.

[0127] (ii) High-risk response detection (sub-step S720).

[0128] The proxy service layer performs high-risk content detection on the placeholder recovery response content. This detection is implemented using a built-in rule engine, which contains a set of predefined high-risk inducement pattern detection rules. These rules are used to identify whether the model output contains content that induces users to disclose sensitive information or perform dangerous operations.

[0129] Specific high-risk patterns include, but are not limited to: requesting API keys or access credentials; requesting environment variable configuration file contents; requesting database access passwords or connection credentials; requesting browser cookies or session tokens; containing dangerous operating system commands (shell commands); and requesting complete internal documentation or system configuration information.

[0130] If the placeholder recovery response does not contain the aforementioned high-risk information, the proxy service layer returns the placeholder recovery response to the client.

[0131] When the response content matches any of the above high-risk events, the proxy service layer first records the event as a risk event in the local database, and then determines the subsequent processing method based on the blocking policy parameter (blockOnHighRiskResponse) in the policy configuration: If the blocking policy parameter is set to mark but not block, the proxy service layer will still restore the response content to the client normally after logging the risk event; If the blocking policy parameter is set to "block," the proxy service layer does not forward the original response from the target upstream endpoint to the client. Instead, it constructs a blocking response locally and returns it to the client with an HTTP 451 status code (unavailable for legal or policy reasons). This blocking response is generated locally by the proxy service layer, rather than originating from the target upstream endpoint; that is, the proxy layer has proactive governance capabilities along the response path.

[0132] The following describes the behavioral feature extraction and log recording (step S800).

[0133] For each request-response interaction that passes through the proxy link, the proxy service layer extracts structured large model behavioral feature data (behavioral feature fields) and writes it to the request log table in the local database.

[0134] The specific meanings of the behavioral characteristic fields are as follows: First Token Time (TTFT): For streaming responses, the first token time is the time interval between sending a request from the proxy service layer and receiving the first response event containing valid content, measured in milliseconds. For non-streaming responses, it records the total delay from sending the request to receiving the complete response body.

[0135] Output text length (outputLength): The number of characters in the text content output by the model.

[0136] Request tokens and response tokens: The number of input tokens and output tokens extracted from the usage field in the response.

[0137] Tokens generated per second (tokensPerSec): A throughput metric obtained by dividing the number of tokens generated by the generation time.

[0138] FinishReason: The reason returned when the model output ends, such as "stop" for normal termination, "length" for reaching the length limit, "tool_calls" for triggering tool calls, etc.

[0139] JSON Validation (jsonValid): For requests that require the model to output in JSON format, record whether the actual output of the model is a valid JSON format.

[0140] ToolCallCount: The number of tool calls initiated by the model in this response.

[0141] Refusal Detection Label: The system performs refusal template matching on the model output to identify whether the model used typical refusal phrases (such as "I can't assist...", "Sorry, I can't...", etc.). If a refusal is detected, it is marked as true.

[0142] Refusal Template Hash: The detected refusal content undergoes text normalization processing (including removing extra whitespace, standardizing capitalization, etc.), and then its hash value is calculated. This hash value does not expose the original content of the refusal response, but can be used for the following analytical purposes: to statistically analyze whether the refusal template has changed along a certain link, whether a new refusal style has emerged, and whether there are signs of suspected model switching. This invention also incorporates the refusal style into the model's behavioral fingerprint.

[0143] Usage Shape Fingerprint: This extracts and normalizes the structural features of usage fields in the response (such as which subfields are included, field naming conventions, etc.) to form a usage shape identifier. Different model service providers or different versions of the model may have differences in the structure of usage fields. This fingerprint can help determine whether the service implementation of the upstream endpoint has changed.

[0144] Error response fingerprint: When an upstream endpoint returns an error response, the structure and key content of the error message are normalized and hashed to form an error fingerprint.

[0145] Tool Name Fingerprint: The tool name fingerprint is formed by sorting and hashing the list of tool names used by the model in tool calls.

[0146] The aforementioned log structure ensures that the system records not just simple interface access logs, but structured profile logs representing the behavior of a large model. This behavioral feature data forms the foundational data source for subsequent drift detection and trustworthiness assessment. The following section details the content regarding drift detection for the target upstream endpoint's behavioral consistency.

[0147] Figure 7 A flowchart of behavior consistency drift detection according to an embodiment of the present invention is shown.

[0148] The drift detection module is used to perform behavioral consistency drift detection on the upstream endpoints of the target. Behavioral consistency drift detection runs continuously as a background scheduled task, aiming to periodically evaluate the behavioral consistency status of each upstream endpoint through active probes. The drift detection module has a built-in fixed probe hint question bank. The design principle of the question bank is: probe hints do not aim to obtain the "correct answer," but rather to obtain stable behavioral feature responses. That is, if the same model gives a highly consistent output to the same probe hint at different time points, and the output changes significantly after the model is replaced, then the probe hint is a valid behavioral feature probe.

[0149] The probe types cover a variety of model capability boundary dimensions, including but not limited to the types shown in Table 1 below.

[0150] Table 1. Probe types in the question bank.

[0151] The aforementioned probe types, by covering multiple dimensions such as format compliance, length control, language switching, rejection style, tool invocation, code generation, and numerical calculation, enable the proxy service layer to capture the behavioral characteristics of the target upstream endpoint from different perspectives. When the target upstream endpoint undergoes model replacement, inference engine change, quantization method change, or service configuration adjustment, the output of the probes in the above different dimensions will change to varying degrees, thereby being identified by the drift detection module.

[0152] Send a preset fixed probe prompt to the upstream endpoint of the target, collect the probe response, compare the current response with the historical output of the same probe for text similarity, and make a drift classification judgment based on the comparison result.

[0153] Step T100: Periodically trigger the drift detection task. Specifically, the timer triggers the proxy service layer to read the policy configuration and determine whether the drift detection module is started. If it is, it iterates through all enabled target upstream endpoints and performs probe request sending and response collection for each endpoint. If it is not started, it skips this round and does not perform detection in this round.

[0154] The specific details of probe request sending and response acquisition (step T200) are as follows.

[0155] After each round of scheduled task is triggered, the proxy service layer iterates through all enabled target upstream endpoints. For each endpoint, it selects one or more probe hints from the probe hint question bank and sends a probe request to the target upstream endpoint in the standard format of the large model compatible interface. The parameter settings of the probe request are kept as deterministic as possible. For example, the temperature parameter is set to 0, and the maximum number of output tokens is set to a small value to reduce the interference of model sampling randomness on output stability.

[0156] The proxy service layer receives the response content returned by the target upstream endpoint for each probe request and extracts the text portion of the response, i.e., the probe response text. When sending probe requests, the proxy service layer also uses the aforementioned compatibility parameter detection and adaptation mechanism to ensure that the probe requests can be correctly received and processed by target upstream endpoints with different implementations.

[0157] The proxy service layer preprocesses the probe response text returned by the upstream endpoint of the target to obtain normalized probe response output text. Specifically: To eliminate spurious differences caused by formatting variations (such as capitalization, punctuation usage, and the number of whitespace characters), the proxy service layer performs the following normalization preprocessing operations on the probe response text before performing text similarity comparison: First, convert all text to lowercase; second, remove punctuation marks from the text; third, remove code block markers (such as Markdown triple backticks and their language identifiers); fourth, compress consecutive whitespace characters (including spaces, tabs, newlines, etc.) into a single space. After the above preprocessing, the normalized probe response output text is obtained.

[0158] Calculate the drift score on the normalized probe response output text (step T300).

[0159] The proxy service layer queries the local data storage module for the previous historical probe output text (also normalized and preprocessed text) of the target upstream endpoint on the same probe cue. If, for a target upstream endpoint, the probe using a certain probe cue is the first probe, then the current output is saved as the initial baseline and no comparison is made.

[0160] If historical output records exist (historical same probe output text), the Dice coefficient (also known as the binary Dice similarity) is used to calculate the text similarity between the current normalized probe response output text and the historical same probe output text. The calculation method of the Dice coefficient is as follows: Each of the two text segments is divided into consecutive bigrams. Let A be the set of bigrams in the current probe response output text, and B be the set of bigrams in the historical output text. Then: Where |A ∩ B| represents the number of identical pairs in the two sets, and |A| and |B| represent the total number of elements in the two sets, respectively. Based on this, divergence is defined as: The divergence value ranges from 0 to 1, where 0 indicates that the current output is almost identical to the historical output, and the closer the value is to 1, the greater the difference in output.

[0161] The technical reasons for choosing the Dice coefficient as a similarity measurement method are as follows: First, the Dice coefficient has low computational complexity and does not rely on external embedding models or additional large models for semantic evaluation, making it suitable for lightweight operation on the agent's local machine; Second, for the specific application scenario of "the same probe prompt, the same endpoint, and comparison of two outputs", the text surface similarity based on tuples can effectively reflect the degree of change in output behavior.

[0162] The proxy service layer determines the drift level based on the calculated divergence value. Specifically, the proxy service layer determines the drift level according to a preset threshold system based on the calculated divergence value: When the divergence is less than the first threshold (default is 0.42), it is considered to be in a normal state, indicating that the current output is highly consistent with the historical output text and the endpoint behavior has not changed significantly. When the divergence is greater than or equal to the first threshold and less than the second threshold (default is 0.62), it is judged as a drift sign, indicating that there is a certain degree of difference between the current output and the historical output text, which may be an early signal of behavioral change; When the divergence is greater than or equal to the second threshold and less than the third threshold (default is 0.75), it is determined to have drifted, indicating that the output difference is relatively significant and the behavior of the upstream endpoint of the target has likely changed. When the divergence is greater than or equal to the third threshold, it is judged as high risk, indicating that the output difference is extremely large and the behavior of the upstream endpoint of the target has undergone a fundamental change, which requires high attention.

[0163] The first to third thresholds can be adjusted through the policy configuration file to adapt to different drift sensitivity requirements in different application scenarios.

[0164] Auxiliary recording mechanism for format expectation verification.

[0165] Each probe tip in the probe question bank also includes format expectation information, such as "output must be valid JSON", "output must contain exactly two list items", "output must contain exactly nine words", and "output must be only the number 408". After each probe is completed, the proxy service layer performs format expectation verification to check whether the actual output of the target upstream endpoint meets the format expectation of the probe tip, and records the verification result as auxiliary metadata in the probe execution log.

[0166] It is important to note that the format expectation check result is recorded only as supplementary reference information and is not directly involved in the drift score calculation. The technical reason for this design is that in actual relay and proxy scenarios, even for the same model, its adherence to format constraints at different times can fluctuate due to factors such as system prompt word configuration, upstream service parameter adjustments, and sampling randomness, leading to a high false positive rate in format expectation checks. Directly including the format expectation check result in the drift score could result in excessive false positives. Therefore, this invention distinguishes text behavior changes (measured by the Dice coefficient) and format constraint failures (recorded by expectation check) as two independent information dimensions: the former serves as the primary signal for drift determination, while the latter serves as supplementary evidence for manual review and comprehensive analysis.

[0167] Probe detection result recording and cooling window mechanism.

[0168] The results of each probe run are written to the probe run record table (sentinel_runs) in the local database. The record includes: endpoint identifier, probe hint identifier, signature of the output text (for subsequent comparison), similarity value, divergence value, delay time, and extracted probe features.

[0169] When the divergence reaches the drift indicator level or higher, the proxy service layer will also generate a drift event and write it to the drift event table (drift_events), and generate a corresponding risk event and write it to the risk event table (risk_events).

[0170] To prevent a large number of duplicate alarms (i.e., alarm storm) when an upstream endpoint of a target remains in a drift state, the proxy service layer implements a cooling window mechanism: for the same upstream endpoint of a target and the same probe prompt, the system will not generate alarms of the same or lower level repeatedly within a preset time window. New alarm records will only be generated when the drift level further increases or the cooling window expires.

[0171] The following details the model fingerprint audit verification under abnormal triggering (step T400).

[0172] When the drift detection module detects that the behavior consistency status of the target upstream endpoint meets the preset trigger conditions (e.g., the drift level reaches "drifted" or "high risk"), or when the administrator manually triggers the audit, the proxy service layer starts the model fingerprint audit module to perform a deeper level of model identity verification on the target upstream endpoint.

[0173] The model fingerprint auditing module uses the B3IT (Border-input Black-box Identification Test) method for model identity verification. B3IT is an existing open-source model fingerprinting method. Its core idea is to utilize a set of carefully designed "border input" cues (short cues indicating that the model has multiple candidate tokens with similar probabilities at the first output token position). By sampling the target upstream endpoint multiple times, the empirical distribution of the first token is obtained. This distribution is then statistically compared with the reference distribution of known models to determine whether the model actually running on the target upstream endpoint is consistent with the declared model. The advantage of this method is that it does not require the target upstream endpoint to provide a logprobs interface; auditing can be completed simply by observing the actual output in a black-box manner. This is suitable for common situations in third-party relay and compatible interface scenarios where the target endpoint does not expose its internal probability information.

[0174] This invention integrates the B3IT method into the model fingerprint auditing module. The specific process is as follows: The first step is to identify the target upstream endpoint and its declared model for the audit, and then locate the corresponding reference distribution baseline. Baseline sources include: automatically matching existing baseline records by declared model name, reading baseline identifiers manually bound in the endpoint configuration, or searching the system's built-in directory of known model fingerprints. If no usable baseline is found, the audit conclusion is directly recorded as "uncertain".

[0175] The second step is to load the set of boundary input prompts from the reference distribution baseline, and collect the first token or first word output by repeatedly sending multiple requests to the target upstream endpoint for each prompt using deterministic parameters (temperature parameter set to 0, output length set to a minimum value), thus forming the empirical output distribution of the target upstream endpoint on each prompt.

[0176] The third step involves calculating the Total Variation Distance (TVD) between the empirical distribution at the upstream endpoint of the target and the baseline reference distribution for each boundary input prompt, and then calculating the p-value through the permutation test to assess the statistical significance of the difference.

[0177] Two sets of data are available: Experimental group samples: the result distribution samples generated by the upstream endpoint of the target in response to boundary prompts; Baseline group samples: samples corresponding to the standard baseline reference distribution.

[0178] A) Merge the experimental group samples and the baseline group samples; B) Randomly redistribute the data into two groups according to the original number of samples in the two groups; C) Recalculate the TVD between the two randomly assigned new groups of data.

[0179] Repeat steps A) to C) M times to obtain a set of random TVDs. Count the number N of all random TVDs that are greater than or equal to observed TVD0, p = N / M.

[0180] The fourth step involves combining the TVD and p-values ​​from all boundary input prompts to generate an audit conclusion: model matching (no significant difference, TVD value below the corresponding threshold, p-value above the corresponding threshold), model mismatch (statistically significant difference, TVD value above the corresponding threshold, p-value below the corresponding threshold), or uncertain (result unclear), along with a confidence score.

[0181] The audit results are written to the fingerprint audit table and risk event table in the local database. When the conclusion is that the model does not match, the severity level of the corresponding risk event is set to a higher level.

[0182] Establishment of the reference distribution baseline: Model fingerprint auditing relies on a usable reference distribution baseline. This invention provides two methods for establishing a baseline: Method 1: Online data collection and establishment. The administrator selects a reliable endpoint of a known model. The proxy service layer samples this endpoint multiple times using all boundary input prompts, statistically analyzes the first token output distribution of each boundary input prompt, and saves it to the fingerprint baseline table in the local database. Method 2: Built-in known model fingerprint directory. The proxy service layer has a built-in set of pre-collected reference distribution information for common large models, which can be directly used as the reference distribution baseline for auditing.

[0183] It's important to note that the B3IT methodology provides probabilistic operational judgments, not absolute identity verification. The audit becomes more challenging when the target upstream endpoint is deliberately imitated or the differences are extremely subtle. However, for common model replacement and service degradation scenarios, this method can typically detect significant distributional differences.

[0184] The key to step T400 is to integrate the B3IT method as a downstream verification link for drift detection into the proxy link, forming a layered triggering mechanism of "lightweight monitoring first, and in-depth verification of anomalies"; and the establishment and matching management of reference distribution baselines (including two sources: online collection and establishment and built-in known model fingerprint directory, and two association methods: automatic matching and manual binding).

[0185] The following section details the comprehensive credibility assessment and risk management (step T500).

[0186] This step integrates the multi-dimensional results from the aforementioned stages (processing of data to be protected, detection of behavioral consistency drift, model fingerprint auditing, and response risk assessment) to generate the overall trustworthiness level of the target upstream endpoint, and executes corresponding risk management strategies accordingly. All risk events are uniformly aggregated into a risk event table for correlation analysis and source tracing.

[0187] Specifically, the credibility determination is based on the following multi-dimensional factors: Dimension 1: Data Protection Event Dimension. This involves analyzing the frequency of data anonymization triggers, the distribution of sensitive data types involved, and the number of session tag interceptions in requests associated with the target upstream endpoint to assess the data exposure risk level of the endpoint's call chain.

[0188] Dimension Two: Behavioral Consistency Drift Dimension. This dimension assesses the overall behavioral stability of the upstream endpoint by considering its drift grading across all probe cues, the mean and maximum drift scores, and the trend of drift level changes (e.g., a continuous slide from "normal" to "drift indication" and then to "drifted").

[0189] Dimension 3: Model Fingerprint Audit Dimension. If a model fingerprint audit has been performed on the target upstream endpoint, the audit conclusion (model match, model mismatch, or uncertainty) and confidence score are taken. The model mismatch conclusion is the most weighted unreliable signal.

[0190] Dimension Four: Response Risk Assessment. This dimension tracks changes in the number of hits, blocking times, rejection template hash values, and usage pattern fingerprints of high-risk content detected in responses returned by upstream endpoints. Changes in rejection template hashes can serve as auxiliary signals that a model may be replaced, and changes in usage pattern fingerprints can serve as auxiliary signals that upstream service implementations may be adjusted.

[0191] Based on the above multi-dimensional comprehensive judgment, the current trust status of the target upstream endpoint is generated, and corresponding risk handling strategies are executed according to the trust status level, including but not limited to: Alarm notification: Push risk alarm information to the administrator through the management console to indicate that the specific endpoint has trust risks; Block forwarding: Suspend the forwarding of subsequent business requests to the target upstream endpoint and remove the target upstream endpoint from the active routes; Restrict forwarding: Reduce the proportion of requests allocated to the target upstream endpoint, or restrict specific types of requests (such as requests containing highly sensitive data) from being sent to this endpoint; Audit logs: Write all related risk events, drift events, audit results, and behavioral characteristic logs into a unified risk event table to form a complete event chain for subsequent source tracing and analysis; Manual review: Mark the endpoint as requiring manual intervention and confirmation, and wait for further verification by the administrator.

[0192] The risk management strategies described above can be customized through strategy configuration files, and different combinations of management strategies can correspond to different trust status levels.

[0193] In addition to the methods described above, other alternative solutions can be adopted for the identification and replacement of protected data, drift detection, model fingerprint auditing, system architecture and deployment, and risk detection and governance on the response side.

[0194] I. Alternative solutions to the data protection identification and placeholder replacement mechanism.

[0195] (1) Alternatives to Sensitive Entity Identification Methods. The aforementioned embodiments employ a rule-driven approach based on regular expressions and pattern matching for identifying the data to be protected. In alternative solutions, the sensitive entity identification method can be replaced by any of the following methods or a combination thereof: Machine learning recognition based on Named Entity Recognition (NER) model: This method uses a pre-trained NER model to identify entities such as names, addresses, organization names, and dates in the request text. Semantic-level sensitive information identification is performed based on a small language model deployed locally. This small model runs locally on the agent and does not send data to the outside. A hybrid recognition method combining keyword dictionaries and regular expressions; The identification method is based on data classification and grading labels. The client marks which fields belong to the data to be protected with predefined labels in the request, and the proxy layer performs the replacement according to the labels.

[0196] The above alternatives do not change the overall closed-loop structure of the present invention, namely "identification → placeholder replacement → local mapping → response recovery", but only the specific implementation method of the identification step is different.

[0197] (2) Alternative to the placeholder generation method. In the aforementioned embodiments, the placeholder is generated by combining a rule type identifier, a counter value, and a random salt value after SHA-256 hashing. In the alternative solution, the placeholder generation method can be replaced by: Use UUID (Universally Unique Identifier) ​​to directly generate placeholder tags, such as tags in the form of `[[RR_550e8400-e29b-41d4-a716]]`; Deterministic encryption methods (such as key-based HMAC) are used to generate placeholders for the original content, so that the same sensitive content generates the same placeholder within the same session, which makes it easier for the upstream model to identify the multiple occurrences of the same entity. Use semantically reserved placeholders to replace opaque markers. For example, replace real email addresses with email addresses that are in a valid format but have fictitious content (such as `user_0001@example.placeholder`) and real phone numbers with numbers that are in a valid format but have fictitious content. This will enable the upstream model to better understand entity types and contextual semantics. Use type-label-based placeholders, such as replacing email with `[EMAIL_1]` and phone with `[PHONE_1]`, so that the placeholders themselves carry entity type information.

[0198] All of the above alternative methods retain the core feature of "reversible replacement and local storage of mapping relationships".

[0199] (3) Alternative storage method for mapping relationships. In the aforementioned embodiments, mapping relationships are stored in the proxy's local memory or local storage. In an alternative, the storage method for mapping relationships can be replaced as follows: Stored in encrypted form in a local database, increasing the security of the mapping table when it is accessed without authorization; In a distributed deployment scenario, the mapping relationship is stored in an internal encrypted cache (such as a memory database like Redis) shared by the proxy cluster to support requests and responses being processed by different proxy instances; By attaching the mapping relationship to the internal metadata of the forwarding request (in encrypted form), the response processing phase can complete the recovery without querying external storage.

[0200] II. Alternatives to drift detection mechanisms.

[0201] (1) Alternatives to the text similarity calculation method. In the aforementioned embodiments, the Dice coefficient (binary Dice similarity) is used to calculate the text similarity between probe outputs. In alternatives, the similarity calculation method can be replaced by any of the following: The Jaccard similarity coefficient is calculated based on the intersection-union ratio of sets of binary or multi-tuples. Edit distance (Levenshtein distance), or normalized edit distance, is calculated based on the minimum number of edit operations required to transform one string into another. Cosine similarity is calculated by converting the text into word frequency vectors or TF-IDF vectors and then calculating the cosine value of the angle between the vectors. After generating text vectors based on a locally deployed lightweight embedding model, the vector distance (such as cosine distance or Euclidean distance) is calculated. Output consistency is measured based on text generation evaluation metrics such as BLEU and ROUGE.

[0202] The above alternative methods do not change the overall drift detection framework of this invention, which is "periodicly collecting output with a fixed probe → comparing with historical output → obtaining a quantitative value of the degree of change → making a graded judgment based on a threshold".

[0203] (2) Alternative to the drift grading threshold. In the aforementioned embodiments, fixed multi-level thresholds (0.42, 0.62, 0.75) were used for grading. In the alternative, the threshold setting method can be replaced as follows: Thresholds are dynamically calculated based on the historical divergence mean and standard deviation of endpoints. For example, "mean plus N times the standard deviation" can be used as the drift judgment boundary, so that the threshold can automatically adapt to the baseline fluctuation level of different endpoints. The trend determination based on the sliding window does not rely on the absolute value of the divergence in a single test, but observes the trend of divergence changes in multiple consecutive tests. When the divergence shows a continuous upward trend, a drift alarm is triggered. Administrators can configure differentiated thresholds based on the business importance and fault tolerance requirements of different endpoints.

[0204] (3) Alternative probe scheduling strategy. The aforementioned embodiments use a fixed-period timed task to trigger probes. In an alternative, the probe scheduling strategy can be replaced with: Adaptive scheduling: Automatically reduce the detection frequency when endpoint behavior is stable (divergence remains low), and automatically increase the detection frequency when drift is detected, in order to dynamically balance resource consumption and monitoring sensitivity; Event-driven scheduling: When abnormal signals such as changes in rejection template hash, usage pattern, or error fingerprint are detected in the behavior feature log, a round of probe detection is triggered immediately, instead of waiting for the next timer cycle; Accompanying scheduling based on request traffic: During every N actual business request processing sessions, an additional probe request is inserted and sent to the upstream endpoint along with the business request.

[0205] (4) Alternative to the probe question bank. The foregoing embodiments use a built-in fixed probe question bank. In the alternative, the probe question bank can be replaced with: Administrators can customize probe hints, allowing the addition of domain-related probes based on specific business scenarios; It supports dynamic expansion of the probe question bank. The system automatically selects the probe prompts that are most sensitive to changes in the behavior of the target endpoint based on the historical results of drift detection and prioritizes their use. It supports loading probe collections from external probe library services to share and update probe resources across multiple agent instances.

[0206] III. Alternatives to the model fingerprint audit mechanism.

[0207] (1) Alternative Audit Methods. The B3IT method was used for model identity verification in the aforementioned embodiments. In alternative solutions, the audit method can be replaced with any of the following: When the upstream endpoint of the target supports logprobs output, the probability vector of the output token is directly collected and statistically compared with the probability vector of the baseline model. This method has higher accuracy but requires the endpoint to open the logprobs interface. Fingerprint recognition is performed based on the differences in the output behavior of specific under-trained tokens using a model; Behavioral fingerprints are compared based on the model's implicit preference patterns (such as wording habits, list style, code indentation habits, etc.) in multi-turn dialogues. Identity determination is performed based on the response characteristics of embedded watermark prompts using a model.

[0208] (2) Alternative to the audit triggering method. In the aforementioned embodiments, auditing is triggered by drift detection anomalies or manually by the administrator. In the alternative, the triggering method can be replaced with: It is automatically triggered periodically to perform a baseline-level fingerprint audit on all endpoints once according to a preset cycle (e.g., daily, weekly); Based on endpoint configuration change triggers, an audit is automatically triggered when an administrator modifies the endpoint configuration (such as changing the base address or changing the API key); Based on the joint triggering of multiple signals from behavioral feature logs, auditing is triggered when two or more of the following conditions are met simultaneously: change in rejection template hash, change in usage pattern, and drift detection anomaly.

[0209] IV. Alternative Solutions for System Architecture and Deployment (1) Alternatives for data storage. The foregoing embodiments use a local SQLite database for data persistence. In an alternative, data storage can be replaced with: Use relational databases such as PostgreSQL and MySQL to support data sharing among multiple agent instances and larger-scale data storage; Use time-series databases (such as InfluxDB and TimescaleDB) to store behavioral characteristic logs and drift events for time-series analysis and trend visualization; Using embedded key-value stores (such as LevelDB and RocksDB) instead of SQLite can achieve higher write throughput in specific scenarios; It adopts a tiered architecture that combines local and remote storage. Hot data is kept locally to ensure low-latency access, while historical data is periodically synchronized to remote storage to support long-term analysis.

[0210] (2) Alternative Deployment Model. In the aforementioned embodiments, the proxy service is deployed locally as a monolithic process. In the alternative solution, the deployment model can be replaced with: Deploying containers (such as Docker containers) facilitates elastic scaling and management in cloud environments or orchestration platforms (such as Kubernetes); It is deployed in the sidecar mode along with the business application process, with the proxy service running as an accompanying process of the business application container; Deployed in a cluster with multiple instances, the multiple agent instances work together through shared storage and load balancing to support high-concurrency scenarios; Deploy some modules of the agent logic (such as drift detection scheduled tasks and fingerprint auditing tasks) in a cloud function or serverless manner to reduce the resident resource consumption.

[0211] (3) Replacement of the compatible interface protocol. In the aforementioned embodiments, the proxy layer exposes an OpenAI-style compatible interface. In the alternative, the compatible interface protocol can be replaced with or extended to: It is also compatible with the interface specifications of other large model service providers, such as the Anthropic Claude interface style and the Google Gemini interface style; Supports gRPC (Remote Procedure Call) protocol as the communication protocol between the client and the agent layer; Supports bidirectional streaming communication via WebSocket persistent connections; The proxy layer implements protocol conversion between various interface specifications, enabling clients to access target upstream endpoints that conform to different interface specifications using only one interface format.

[0212] V. Alternative Solutions for Response-Side Risk Detection and Management (1) Alternatives to high-risk detection methods. The aforementioned embodiments employ a built-in rule engine for high-risk induced pattern detection. In alternative solutions, the detection method can be replaced with: Risk category determination is based on a locally deployed small text classification model, supporting more complex semantic-level risk identification. Based on a configurable external risk assessment interface, the response text is sent to the organization's internal content security auditing service for testing; A hybrid detection approach that combines rule engines and machine learning models, where rule engines are used to quickly match known patterns and machine learning models are used to identify new risk patterns; Administrators can customize detection rules and templates to adapt to the compliance requirements of specific industries or business scenarios.

[0213] (2) Alternative blocking strategies. In the aforementioned embodiments, the blocking strategies are either "marking without blocking" or "directly blocking and returning HTTP 451". In alternative solutions, the blocking strategy can be expanded to: Partial blocking: Only high-risk content segments in the response are masked or replaced, rather than blocking the entire response, thus preserving the safe parts of the response; Degradation replacement: Replace high-risk responses with locally preset safe alternatives instead of returning error status codes; Secondary confirmation: The high-risk response will be temporarily stored and the administrator will be notified. The administrator will then confirm and decide whether to allow or block the response. Tiered blocking: Different levels of intervention are taken based on the severity of the risk. Low-risk cases are simply marked, medium-risk cases are marked with a warning and then allowed to pass, and high-risk cases are blocked directly.

[0214] Besides proxy scenarios for large model-compatible interfaces, the technical solution of this invention can also be applied to the following uses: (1) Compliance audit of large-scale model services within enterprises. After an enterprise purchases or builds multiple large-scale model services, it can deploy the system of this invention as a unified compliance audit gateway. Through the identification of data to be protected and the placeholder replacement mechanism, it ensures that employees will not disclose sensitive internal information (such as customer data, trade secrets, access credentials, etc.) to external model services when using large-scale model services; through behavioral feature logs and drift detection mechanisms, it continuously audits the consistency of behavior of each model service endpoint to ensure that the model services provided by the supplier meet the service level agreed in the contract.

[0215] (2) Service quality monitoring of large model service providers. Purchasers of large model services can use the drift detection and model fingerprint audit functions of this invention to continuously verify the model services declared by the supplier, identify whether the supplier has downgraded, replaced or reduced the capabilities of the model during operation, and provide technical basis for performance evaluation of service level agreement (SLA).

[0216] (3) Debugging and regression testing assistance during the development phase of large model applications. During the development and testing phase of large model applications, developers can use the behavioral feature logs and drift detection functions of this invention to monitor the output behavior changes of the dependent model services during different version iterations, and assist in discovering application function regression problems caused by upstream model updates.

[0217] (4) Unified security gateway for multi-model services. In scenarios where multiple large model service providers are used simultaneously within an organization, this invention can be deployed as a unified security gateway, implementing consistent data protection policies for all outbound requests, consistent risk detection policies for all inbound responses, and independent behavior monitoring and trust assessment for each endpoint, thereby achieving unified security governance in a multi-model service environment.

[0218] (5) Compliance access layer for large models in sensitive industries. In industries such as finance, healthcare, and government affairs, which have high requirements for data security and service credibility, this invention can serve as a compliance access layer between business systems and external large model services, meeting regulatory compliance requirements such as sensitive data not leaving the domain, traceability of model services, and auditability of abnormal behavior.

[0219] The method and system of this invention overcome the contradiction between protection and availability in existing data protection schemes while achieving the security of the data to be protected, the availability of upstream processing, and the recoverability of response results. Existing second-type schemes (data security and observation schemes for LLM call chains) typically employ methods such as direct interception, static masking, or irreversible desensitization when handling sensitive information in requests. Direct interception prevents requests containing sensitive information from being processed by upstream endpoints, while static masking and irreversible desensitization destroy the semantic structure of the request content, affecting the processing effect of the upstream large model. Furthermore, after the response is returned, the portion of the processing result containing sensitive information cannot be restored to its original content, reducing the actual usability of the response result for the client. This invention employs a reversible placeholder replacement mechanism to replace sensitive entities with opaque placeholders containing random salt values ​​and hash digests before the request leaves the server, ensuring that the original sensitive data never leaves the local proxy service layer. Simultaneously, because the placeholders retain the semantic location information "an entity exists here" within the request text, the large model of the target upstream endpoint can still complete processing tasks based on the semantic skeleton. After the response is returned, the system reverse-engineers the placeholders back to their original content based on locally stored mapping relationships (including raw token-compatible restoration), allowing the client to ultimately obtain a usable response containing complete original information. Therefore, this invention achieves an effective balance between data protection, upstream processing availability, and response recoverability, rather than forcing trade-offs between protection strength and availability as in existing technologies.

[0220] The method and system of this invention achieve continuous and proactive monitoring of the consistency of upstream endpoint behavior, filling the gap in the identification of endpoint trust status in existing proxy gateway solutions. The core objective of existing first-type solutions (large model-compatible interface proxy or gateway solutions) is unified access and request forwarding. Their operational monitoring is typically limited to traditional API operation and maintenance metrics such as availability, latency, throughput, and error rate. These metrics only reflect the basic operational status of the endpoint and cannot identify whether the upstream endpoint has experienced issues such as model replacement, inference engine changes, quantization method changes, or output behavior drift that do not affect interface availability but substantially alter service capabilities. This invention uses a drift detection module (Sentinel) to continuously run proactive probe tasks in the background. It periodically sends probe requests to each upstream endpoint using a fixed probe question bank covering multiple model capability boundary dimensions. The current probe output is compared with historical outputs of the same probe based on the Dice coefficient, and the divergence is calculated and multi-level threshold judgments are performed. This method does not rely on externally embedded models or additional large models, is lightweight and implemented only locally on the proxy, and employs a design that separates format expectation verification and drift score to reduce false positive rates. A cooling window mechanism is used to avoid alarm storms. Therefore, this invention enables the proxy layer to have continuous, proactive, and lightweight monitoring capabilities for the behavior and status of the target upstream endpoint, and can still detect substantial changes in endpoint behavior even when traditional operation and maintenance indicators show "health".

[0221] The method and system of this invention implement a layered triggering mechanism of "lightweight monitoring first, then in-depth verification of anomalies," improving the accuracy of endpoint trustworthiness judgment and resource utilization efficiency. Existing third-category solutions (model fingerprinting, endpoint stability, or identity analysis solutions) can analyze model identity and behavioral stability from an academic or detection perspective, but these solutions mostly operate as independent detection tools, consuming significant sampling resources and typically not integrated with actual business proxy links. Continuously performing high-overhead fingerprint audits on each endpoint would significantly increase the request burden on upstream endpoints and system operating costs. This invention adopts a layered triggering design: under normal conditions, only lightweight drift detection is run (only a small number of probe requests are sent per round); only when the drift level reaches a preset triggering condition is further initiated to initiate model fingerprint auditing based on the B3IT method to perform multiple sampling and statistical comparisons of the target endpoint. This layered mechanism allows the system to maintain low resource consumption during the majority of endpoint behavior cycles, investing in in-depth verification resources only when needed, achieving a balance between the accuracy of trustworthiness judgment and system operating efficiency.

[0222] This invention integrates data protection, proxy forwarding, response recovery, behavior monitoring, identity verification, and risk handling into a single proxy link, overcoming the problem of existing technologies where each capability is independent and difficult to form a closed loop. In existing technologies, capabilities such as compatible interface proxying, sensitive information processing, call observation, model behavior analysis, and anomaly alerting are typically implemented by different solutions, lacking a unified processing link and collaborative triggering relationship. For example, data protection solutions do not consider endpoint trustworthiness, endpoint monitoring solutions do not process sensitive data in requests, and model identity analysis solutions are not integrated with the proxy forwarding process. This invention unifies these previously scattered technical capabilities into a single proxy processing link: when a request enters, policy interception and placeholder replacement of protected data are performed; during forwarding, authentication processing and compatibility adaptation are performed; when a response returns, placeholder marker restoration and high-risk content detection and blocking are performed; throughout the runtime, behavior consistency drift detection is continuously performed; when an anomaly is triggered, model fingerprint audit verification is performed; and finally, the results from all dimensions are comprehensively judged and a unified risk handling is performed. All the above links form a complete closed-loop control process around the same upstream endpoint, and risk events are uniformly aggregated into a single data table for correlation analysis and source tracing. Compared to the single-point governance approach of existing technologies, the integrated closed-loop architecture of this invention can form information transmission and collaborative triggering between various links, making the overall governance effect better than the effect of simply superimposing the parts.

[0223] This invention achieves all governance functions while keeping the client's calling method completely unchanged, reducing deployment and integration costs. Existing solutions often require modifications to the client's calling code, interface parameters, SDK version, or deployment architecture when implementing data protection, access control, or anomaly detection functions. The proxy service layer of this invention exposes standard large-model compatible interfaces (including chat completion, response generation, and model list query interfaces). Clients only need to point the interface base address to the proxy service layer address to access all the governance capabilities of this invention without modifying any calling code or interface parameters. For existing business applications already developed using large-model compatible interfaces, the integration cost of this invention is extremely low, requiring no adjustments to business logic, SDK dependencies, or deployment processes. Furthermore, the proxy service layer uses a local SQLite database for storage, eliminating reliance on external databases or cloud services. The entire system can be deployed as a monolithic service, suitable for use in private and restricted network environments.

[0224] The method and system of this invention possess proactive governance capabilities along the response path, rather than merely providing passive alerts or post-event logging. Existing second-type solutions largely limit their observation and security capabilities to logging, event tagging, and alert notifications—notifying administrators after a risk is detected, but the request-response process itself remains unaffected, and the risky content still reaches the client. This invention incorporates a high-risk inducement pattern detection rule engine into the response path. This engine can identify content in the model output at the proxy layer that induces users to disclose sensitive information or perform dangerous operations, and proactively constructs a blocking response locally on the proxy to return to the client based on policy configuration, rather than simply transmitting the high-risk content. This means that the proxy layer possesses proactive intervention and governance capabilities along the response path, intercepting risky content before it reaches the client, elevating the response from passive "post-event discovery" to proactive "in-event blocking."

[0225] The behavioral feature logs of this invention significantly surpass the richness of existing proxy solutions, providing a stronger data foundation for long-term operational analysis and anomaly tracing. Existing first-type solutions typically record request logs containing general API access information such as routing paths, status codes, latency, and request sizes. This invention additionally extracts structured features from each request-response interaction, targeting the behavior of the large model, including first token time, rejection template hash value, tool name fingerprint, usage pattern fingerprint, error response fingerprint, JSON format validity, termination reason, and streaming event statistics. These feature fields enable the system to build endpoint profiles from the perspective of model output behavior, rather than simply recording access information at the network and interface levels. For example, changes in the rejection template hash value can serve as an auxiliary signal that the model may be replaced, changes in the usage pattern fingerprint can indicate adjustments to upstream service implementations, and changes in the tool name fingerprint can reflect differences in the model's tool invocation capabilities. This rich behavioral feature data not only provides more dimensional signal inputs for drift detection and trustworthiness determination but also provides administrators with a data foundation far exceeding that of existing solutions for long-term operational trend analysis and anomaly tracing.

[0226] This invention unifies the following capabilities, originally belonging to different technical solutions, into a single proxy processing chain oriented towards a large model-compatible interface: request preprocessing and policy interception (session tag interception, local exclusive content stripping), identification and placeholder replacement of data to be protected, proxy forwarding, streaming and non-streaming response processing and recovery, high-risk content detection and proactive blocking on the response side, behavioral feature extraction and structured log recording, periodic behavioral consistency drift detection, model identity verification under abnormal triggering, and comprehensive trust determination and risk handling. These links form a complete closed loop around the same upstream endpoint, from request entry to continuous monitoring and abnormal handling throughout the runtime cycle. Clients only need to point the interface base address to the proxy service to access it, without changing their original calling method. This integrated architecture is the core innovation that distinguishes it from existing technologies where each capability is implemented independently.

[0227] While some embodiments of the present invention have been described in this application, those skilled in the art will understand that these embodiments are merely illustrative. Numerous variations, alternatives, and improvements will arise in those skilled in the art under the teachings of this invention without departing from its scope. The appended claims are intended to define the scope of the invention and thereby cover methods and structures within the scope of the claims themselves and their equivalents.

Claims

1. A proxy-based endpoint trusted monitoring and data protection method for large model-compatible interfaces, characterized in that, include: The proxy service layer receives the original request sent by the client according to the large model compatible interface format, and determines the corresponding target upstream endpoint from multiple upstream endpoints; The proxy service layer identifies the data to be protected in the original request, replaces the data to be protected with placeholders, obtains the de-identification request, and saves the replacement mapping table; The proxy service layer constructs the HTTP request header and combines it with the de-identified request to form the final upstream request, which is then sent to the target upstream endpoint. The proxy service layer receives the raw response returned by the target upstream endpoint, performs streaming or non-streaming response processing, and obtains the response output text. The proxy service layer replaces the placeholders in the response output text with the data to be protected according to the replacement mapping table, obtains the placeholder recovery response content, and forwards it to the client, completing a request-response interaction; as well as During each request-response interaction, the drift detection module of the proxy service layer periodically performs drift detection on the behavior consistency of the target upstream endpoint, including: sending a preset fixed probe prompt to the target upstream endpoint, collecting probe responses, calculating the text similarity and divergence between the current response and the historical output text of the same probe, and making a drift classification judgment based on the divergence calculation results.

2. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, Identify the data to be protected in the original request, replace the data with placeholders, and obtain the de-identified request, which includes: Iterate through all text fields in the original request; perform multi-type sensitive entity rule matching on each text field to filter out sensitive entities; generate a unique placeholder for each matched sensitive entity, wherein generating the placeholder includes: concatenating the type identifier of the current matching rule, the counter value of the current replacement operation, and a randomly generated salt value, calculating a 256-bit digest using a secure hash algorithm, taking the first few characters of the digest, and combining them with a fixed prefix and counter number to form the placeholder; The replacement mapping table is stored in the local data storage module. Each record in the replacement mapping table contains: placeholder, original sensitive entity, matching rule type, and position information in the original request. The de-identification operation will be recorded as a risk event in the risk event table of the local data storage module. The proxy service layer replaces the placeholders in the response output text with the data to be protected based on the replacement mapping table, resulting in a placeholder recovery response that includes: The proxy service layer scans the response output text and identifies placeholders in the response output text through full format matching or naked token compatibility matching. The proxy service layer restores the placeholder markers to the corresponding sensitive entities based on the replacement mapping table.

3. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, The drift detection module of the proxy service layer periodically performs consistency drift detection on the upstream endpoint behavior of the target, including: The drift detection module has a built-in probe-based question bank; The timer triggers the proxy service layer to read the policy configuration, determine whether the drift detection module is started, and if so, it iterates through all enabled target upstream endpoints, performing probe request sending and response collection for each target upstream endpoint, including: Select one or more probe hints from the probe hint question bank and send a probe request to the target upstream endpoint in the standard format of the large model compatible interface; The proxy service layer receives the response content returned by the target upstream endpoint for each probe request and extracts the text portion of the response, i.e., the probe response text; The proxy service layer preprocesses the probe response text to obtain normalized probe response output text: converting all text to lowercase; removing punctuation marks from the text; removing code block markers from the text; and compressing consecutive whitespace characters into a single space. The text similarity between the current normalized probe response output text and the historical output text of the same probe is calculated, and the divergence is calculated based on the text similarity calculation results. Drift classification is determined based on the divergence calculation results, including: If the calculated divergence is less than the first threshold, it is considered to be in a normal state. When the divergence calculation result is greater than or equal to the first threshold and less than the second threshold, it is judged as a sign of drift; When the divergence calculation result is greater than or equal to the second threshold and less than the third threshold, it is determined that it has drifted; When the divergence calculation result is greater than or equal to the third threshold, it is judged as high risk.

4. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 3, characterized in that, Each probe hint in the probe question bank also includes format expectation information. When the proxy service layer performs format expectation verification after each probe request sending and response collection, it checks whether the response output of the target upstream endpoint meets the format expectation of the probe hint and records the format expectation verification result as auxiliary metadata in the probe running record.

5. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 3, characterized in that, Also includes: When the drift detection module detects that the behavior consistency status of the target upstream endpoint meets the preset trigger conditions, or when the administrator manually triggers the audit, the proxy service layer starts the model fingerprint audit module to perform model identity verification on the target upstream endpoint.

6. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 5, characterized in that, The preset triggering condition is met when the drift classification judgment reaches the drift level or the high-risk level. The model identity verification is performed by the model fingerprint auditing module using the B3IT boundary input black-box identification test method, including: Identify the target upstream endpoints and their declarative models for the audit, and find the corresponding reference distribution baselines; Load the set of boundary input prompts in the reference distribution baseline, and collect the first token or first word output by repeatedly sending multiple requests to the target upstream endpoint for each prompt using deterministic parameters, forming the empirical output distribution of the target upstream endpoint on each prompt; For each boundary input prompt, calculate the total variation distance between the empirical distribution of the upstream endpoint of the target and the baseline reference distribution, and calculate the p-value through a permutation test to assess the statistical significance of the difference; By combining the TVD and p-values ​​from all boundary input prompts, an audit conclusion is generated, and a confidence score is given. The audit conclusion includes: model matching, model mismatch, or uncertainty. The sources of the reference distribution baseline include: establishing it through online collection, including: the administrator selecting an endpoint of a known model, the proxy service layer sampling the endpoint multiple times using all boundary input prompts, statistically analyzing the first token output distribution of each boundary input prompt, and saving it to the fingerprint baseline table in the local database; or The proxy service layer has a built-in set of pre-collected large model reference distribution information as a reference distribution baseline.

7. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, Determining the corresponding target upstream endpoint from multiple upstream endpoints includes: After parsing the model name field in the original request, the proxy service layer calls the model name matching routing method in the endpoint storage module to find the upstream endpoint that best matches the model name in the current request among all configured upstream endpoints as the target upstream endpoint. The model name matching routing method uses a multi-level matching scoring mechanism, with the following priorities from highest to lowest: First priority, exact match: compare the model name in the request with the model name declared in the configuration of each upstream endpoint. If they are exactly the same, the model name is directly used as the target upstream endpoint. Second priority, model name suffix matching: Determine whether the model name in the original request is the suffix of the endpoint declaration model name. If so, it is considered a suffix match. The third priority is substring inclusion matching: determining whether the model name in the original request is contained within the endpoint declaration model name; Fourth priority, despeccable matching: compare the model name in the original request and the endpoint declaration model name after removing the separator; Fifth priority, token hit matching: split the model name and endpoint declaration model name in the original request into multiple tokens by the separator, calculate the hit rate of the common tokens between the two model names, and take the one with the highest hit rate.

8. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, Before replacing the placeholder data to be protected, the process also includes: performing a session tag interception check on the original request; if any session tag carried in the current request matches a blacklist item, an error response is returned to the client; and The original request is stripped of its local content, and the local content fragments are replaced with placeholder prompts. Before forwarding the placeholder recovery response content to the client, the following is also included: The proxy service layer performs high-risk content detection on the placeholder recovery response content. If the detection passes, the placeholder recovery response content is directly returned to the client. If high-risk content is detected, the risk event is recorded in the local database, and the handling method is determined according to the blocking policy parameters in the policy configuration, and corresponding processing is performed, including: If the blocking policy parameter is set to mark but not block, the proxy service layer will still return the placeholder recovery response content to the client after logging the risk event; If the blocking policy parameter is set to block, the proxy service layer will not forward the original response of the target upstream endpoint to the client, but will construct a blocking response locally in the proxy service layer and return it to the client with an HTTP 451 status code. The built-in rule engine performs high-risk content detection on the placeholder recovery response content. The built-in rule engine contains a set of predefined high-risk inducement pattern detection rules to identify whether there is content in the model output that induces users to disclose sensitive information or perform dangerous operations.

9. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, The proxy service layer receives the raw response returned by the target upstream endpoint, performs streaming response processing, and obtains the response output text, which includes: When the client declares a streaming response in the original request, and the response content type returned by the target upstream endpoint is a streaming response type, the proxy service layer enters the streaming response processing path, including: Initialize streaming statistics variables, including: first token timestamp, streaming event counter, payload character accumulator, output text accumulation buffer, tool call counter, usage pattern record variable, and termination reason record variable; The proxy service layer reads events from the streaming responses returned by the upstream endpoints one by one. For each event, the proxy service layer performs two operations simultaneously: it transmits the event data to the client's response connection as is, ensuring the client receives the streaming response output in real time; and it parses the event payload locally, including: the proxy service layer performs the following analysis on the payload of each event: if the event is the first event containing valid text content, it records the time difference between the current time and the request sending time as the first token time; it extracts the incremental text content fragments from the event payload and appends them to the output text accumulation buffer; it detects and accumulates tool call-related fragments in the event payload; it accumulates the event count and payload character count to the corresponding counters; if the event payload contains a usage field, it records the usage morphology characteristics; if the event payload contains an end reason field, it records the end reason value. After the streaming response output content has been transmitted, the proxy service layer performs placeholder restoration processing on the complete output text spliced ​​in the output text accumulation buffer.

10. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, The proxy service layer receives the raw response returned by the target upstream endpoint, performs non-streaming response processing, and obtains the response output text. When the client's original request does not declare streaming or the response returned by the target upstream endpoint is in standard JSON format, the proxy service layer enters the non-streaming response processing path, including: The proxy service layer reads the entire response returned by the target upstream endpoint as text, thus obtaining the response text; The received response text is parsed using JSON to obtain the response content.

11. The proxy endpoint trusted monitoring and data protection method for large model compatible interfaces according to claim 1, characterized in that, It also includes: for each request-response interaction, the proxy service layer extracts large model behavior feature data and writes it to the request log table in the local database; the feature fields in the request log table include the first token time, output text length, number of tokens consumed in the request and the number of tokens generated in the response, number of tokens generated per second, termination reason, JSON format validity, number of tool calls, rejection detection flag, rejection template hash value, usage pattern fingerprint, error response fingerprint, and tool name fingerprint; Before replacing the placeholder data to be protected, the following steps are also included: The original request is intercepted and checked for session tags. If any session tag carried in the current request matches a blacklist item, an error response is returned to the client. The original request is stripped of its local content, and the local content fragments are replaced with placeholder prompts.

12. A proxy-based endpoint trusted monitoring and data protection system for large model-compatible interfaces, characterized in that, Deployed between the client and multiple upstream endpoints, including: The transparent protocol proxy module is configured to perform the following actions: receive the original request sent by the client in a large model-compatible interface format, parse the model name field in the original request to determine the corresponding target upstream endpoint; and route the de-identified request processed by the local data protection module to the corresponding target upstream endpoint. The local data protection module is configured to perform the following actions: before the original request is sent to the target upstream endpoint, perform local-specific content stripping, session tag interception and inspection, and decoy tag injection operations on the original request; identify the data to be protected in the original request, replace the data to be protected with placeholder tags, generate a de-identified request, and store the replacement mapping relationship between the placeholder tags and the data to be protected in the local data storage module; and receive the response result returned by the target upstream endpoint, and replace the placeholder tags in the response result with the original data to be protected according to the replacement mapping relationship. The drift detection module is configured to perform the following actions: during each request-response interaction, periodically and actively probe the upstream endpoint of the target with a fixed probe cue set, calculate the text similarity and divergence between the current probe output and the historical output text of the same probe, and make a drift classification judgment based on the divergence calculation results, thereby identifying whether there is behavioral drift at the upstream endpoint of the target. The model fingerprint audit module is configured to perform the following actions: when the drift detection module detects behavioral drift at the target upstream endpoint or when the administrator manually triggers the audit, it performs model identity verification on the target upstream endpoint.

13. The proxy endpoint trusted monitoring and data protection system for large model compatible interfaces according to claim 12, characterized in that, It also includes a behavior feature extraction module, which is configured to perform the following actions: extract structured large model behavior feature data from the response returned by the target upstream endpoint and write it to the request log table of the local data storage module.

Images