Black box tokenizer adaptive boundary probe and word difference positioning method and system

Abstract

The present application is suitable for computer natural language processing, software testing and word meter calibration technical field, and provides a black box word segmenter adaptive boundary probe and word difference positioning method and system, the method comprises the following steps: obtaining a candidate word segmenter configuration and a limited probe domain, constructing a state diagram and a predicted response signature function; deriving an equivalent class according to the predicted response, generating a probe from the distinguishable candidate state edge, and determining the target probe after embedding the message envelope; calling the black box interface, comparing the measured response with the predicted response to update the candidate or identify the unknown; generating the minimum distinguishable probe pair and positioning the earliest difference node; when the stop condition is met, output the evidence mapping. The present application is suitable for the verification of the black box word segmenter which is inaccessible to the internal product, and can trace and locate the word behavior difference.

Description

Technical Field

[0001] This invention relates to the fields of computer natural language processing, software testing, and lexical measurement and calibration, specifically to a black-box word segmenter adaptive boundary probe and lexical difference localization method and system. Background Technology

[0002] Tokenizers convert raw text or byte sequences into ordered sequences of token identifiers and are one of the core components of natural language processing models. Different tokenizer implementations may employ various processing mechanisms such as Unicode normalization, pre-segmentation, byte pair encoding (BPE), WordPiece longest match, unigram path selection, special token insertion, and message template serialization. For open-source or locally deployed tokenizers, technical personnel can understand their behavior by examining the vocabulary, merging rules, and configuration files; however, for closed-source commercial model services (i.e., black-box tokenizers), clients typically can only submit requests through application programming interfaces and observe limited information in the returned results, such as the total number of tokens, token identifiers, byte boundaries, or usage fields. The server may also contain hidden templates or special token processing logic that is not visible to the client. Existing technologies, such as CN114912089A, involve algorithm error assessment; US20260056771A1 involves word segmenter verification; and US20260058997A1 involves token budget verification. However, none of these solutions can solve the technical problems of actively generating test cases, gradually narrowing down the candidate range, and locating the root cause of differences under black-box conditions.

[0003] Specific defects in existing technologies include: First, testing using fixed natural corpora, random fuzzy input, or artificial boundary test sets typically does not establish a correspondence between test cases and the specific mechanisms within the segmenter (such as a certain step of merging or normalization). Test items lack a mapping to the state edges of the candidate segmenter's internal mechanisms. When the number of candidates is large, full calls increase network transmission, processing latency, and the number of interface calls. When multiple candidates output the same value on the tested corpus, it is difficult to determine whether they are truly equivalent or have not yet triggered the difference edge. Second, if the black-box return value does not belong to the prediction result of any preset candidate, existing technologies usually forcibly assign it to the closest candidate based on edit distance or numerical distance, leading to incorrect unique configuration conclusions. At the same time, directly writing server-side hidden fields into the client message envelope can confuse observed facts with inferred hypotheses, further affecting the accuracy of the test. Third, existing methods lack a technical mechanism to connect candidate mechanism state edges, real message serialization envelopes, response signature derived equivalence classes, unknown candidate detection, minimum distinguishing probe pairs, and verifiable stopping conditions into a closed loop. This makes it impossible to output credible limited conclusions rather than forged unique configurations when the call budget is limited or the interface only returns the total number of tokens.

[0004] Therefore, in view of the above situation, there is an urgent need to provide a method and system for adaptive boundary probes and word difference localization of black-box word segmenters to overcome the shortcomings in current practical applications. Summary of the Invention

[0005] The purpose of this invention is to provide a method and system for adaptive boundary probes and word difference localization in black-box word segmentation, effectively solving the problems mentioned in the background art.

[0006] This invention is implemented as follows: the black-box word segmenter adaptive boundary probe and lexical difference localization method, executed by at least one processor, includes: Obtain multiple candidate segmenter configurations and limited probe domains, construct a mechanism-specific lexicalization state graph for each candidate segmenter configuration, and establish a prediction response signature function for the candidate segmenter configuration to the probe; Based on the equivalence relation that the predicted response signatures generated by each candidate word segmenter for the executed probe are the same, the candidate behavior equivalence classes are dynamically derived. Candidate boundary probes are generated from candidate state edges that enable at least two candidate segmenters in the same current candidate behavior equivalence class to produce different predicted response signatures, and each candidate boundary probe is embedded into the client message serialization envelope. Determine the target probe from the candidate boundary probes; The target probe is used to call the black-box word segmenter, obtain the interface return, form a measured response signature from the valid response, and compare the measured response signature with the candidate predicted response to update the candidate word segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state. Generate minimum distinguishing probe pairs, and determine the earliest distinguishable processing node based on the response differences of the minimum distinguishing probe pairs; Stop when the preset stopping conditions are met, and output the corresponding evidence mapping.

[0007] As a further aspect of the present invention: the state edges of the lexicalized state graph at least record the previous state, the triggering context, the candidate output, and the successor state; The mechanism-specific lexical state graph includes at least one of the following: For a byte-pair encoding word segmenter, the state is composed of the current symbol unit sequence, adjacent candidate pairs, and merge ranking, and the state edges are composed of performing the selected merge and updating the adjacent candidate pairs. For the WordPiece word segmenter, the state is composed of the vocabulary prefix node, the current input position, the longest matching candidate and the unknown word backoff flag, and the state edge is composed of the execution of the longest match or the unknown word backoff. For the Unigram segmenter, the state is composed of input position, candidate lexical arc, candidate cost, cumulative path value and parallel decision marker, and the state edge is composed of selecting candidate lexical arc and updating the optimal path. Generating candidate boundary probes from the candidate state edges includes: generating candidate boundary probes from candidate state edges and their triggering contexts that enable at least two candidate word segmenters in the same current candidate behavior equivalence class to generate different predicted response signatures; specifically, it includes: inversely calculating the shortest preceding byte sequence that makes the target state edge reachable, connecting the target byte sequence that makes the target state edge valid and the shortest following byte sequence that keeps the subsequent response observable. The predicted response signature and the measured response signature include at least the total number of tokens, and when the interface is available, further include at least one tokenized observable among ordered token identifiers, UTF-8 byte boundaries, special token positions, probability values, or logarithmic probability values; the response signature is normalized and encoded according to a predetermined field order, and the source span set, target span, and mapping type are irreversibly normalized and stored. The client message serialization envelope is the client real message serialization envelope. The client real message serialization envelope consists of at least one of the following: client request payload bytes, character encoding, field order, escape rules, role sequence, and tool definition that are observable by the client before transmission. Unobservable server-side hidden fields are not treated as known fields. For server-side hidden templates or hidden special terms, they are only recorded as candidate hypotheses with support probes and observability constraints. Determining the target probe from the candidate boundary probes includes: calculating the predicted response partitions formed by each candidate boundary probe with respect to the current candidate segmenter configuration, and determining the target probe from the candidate boundary probes with minimizing the maximum number of remaining candidates in the predicted response partition as the selection objective; the candidate weights are derived from a prior distribution with version identifiers, and the default weights of the current candidates are equally normalized when the prior distribution does not exist; within the limited probe set, the distance between two candidates is defined as the proportion of probes whose predicted response signatures are different; the equivalence class diameter of a predicted response partition is defined as the maximum distance between any two candidates in that partition; for each candidate boundary probe, the maximum number of remaining candidates, the candidate weight of the maximum partition, and the maximum equivalence class diameter are calculated, and the target probe is determined in lexicographical order of first minimizing the maximum number of remaining candidates, then minimizing the candidate weight of the maximum partition, and then minimizing the maximum equivalence class diameter.

[0008] As a further aspect of the present invention: comparing the measured response signature with the candidate predicted response to update the candidate segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state, specifically including: For deterministic interfaces, a measured response signature is generated from valid responses; for black-box interfaces marked as nondeterministic, the same target probe is executed according to preset repeated confirmation conditions to generate a measured response distribution signature. Compare the measured response signature with the predicted response partition, or compare the measured response distribution signature with the predicted response distribution partition; When a corresponding prediction partition is hit, the candidate segmenter configuration and candidate behavior equivalence class are updated. When no corresponding prediction partition is hit, an unknown candidate state is generated. The process of generating the unknown candidate state includes: confirming that the interface returns a response that conforms to a predetermined response structure, that the request idempotent identifier is consistent with the target probe, and that the applicable duplicate confirmation conditions are met; after meeting the above conditions, determining that the measured response signature or measured response distribution signature is different from all candidate predicted response signatures or predicted response distribution signatures; saving the original bytes, envelope, interface return text, and time information of the target probe; and ensuring that the unknown response is not forcibly merged into the nearest predicted response partition.

[0009] As a further aspect of the present invention: generating the minimum distinguishing probe pair includes: reversing the common pre-context from the first separated state edge of the two candidate state paths, constructing a first probe and a second probe that are identical except for a controlled byte, a controlled byte span, or a message serialization envelope field; when the measured response signature difference between the first probe and the second probe is consistent with the predicted difference of the first separated state edge, the processing node corresponding to the state edge is determined as the earliest distinguishable processing node.

[0010] As a further aspect of the present invention: when a black-box call results in network failure, rate limiting, timeout, error page, or truncated response, an idempotent call identifier is established for the same probe and the state before candidate update is maintained, and the call is repeated according to the backoff strategy; network failure, rate limiting, timeout, error page, or truncated response are only recorded as abnormal evidence without updating the candidate set or generating an unknown candidate state; when the same probe generates multiple valid responses, a measured response distribution signature is formed based on the response signature frequency and confidence interval, and compared with the predicted response distribution signature formed by each candidate configuration under the same repeated call conditions.

[0011] As a further aspect of the present invention: the preset stopping condition includes at least one of the following: the candidate segmenter configuration converges to the unique candidate segmenter configuration, the remaining candidates have an indistinguishable proof of response signatures within the defined probe domain, the preset call budget is reached, and the unknown candidate state is generated; when the black-box interface only returns the total number of tokens used, the total number of tokens used constitutes a count-level response signature, and the count-level candidate behavior equivalence class and observability constraint are output; the indistinguishable proof includes the generation rules of the defined probe domain, the enumerated probes or state edge set, the response signature matrix of the remaining candidates, and the integrity digest.

[0012] This invention also provides a black-box word segmenter adaptive boundary probe and lexical difference localization system, which performs the method described above, including: The candidate modeling module is used to obtain multiple candidate word segmenter configurations and limit probe domains, construct a mechanism-specific lexicalization state graph, and establish a predictive response signature function; The equivalence class derivation module is used to dynamically derive candidate behavior equivalence classes based on the relationship between the predicted response signatures of the executed probes and the candidate word segmenters. The probe and envelope module is used to generate candidate boundary probes from candidate state edges that enable at least two candidate word segmenters in the same current candidate behavior equivalence class to generate different predicted response signatures, and to embed the candidate boundary probes into the client message serialization envelope. A probe selection module is used to determine the target probe from the candidate boundary probes; The response validity and update module is used to call the black-box word segmenter and verify the validity of the return, form a measured response signature from the valid response, and compare the measured response signature with the candidate predicted response to update the candidate word segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state. The differential localization module is used to generate minimum distinguishing probe pairs and determine the earliest distinguishable processing node based on response differences; It also includes a stop and evidence module, which is used to stop when a preset stop condition is met and output the corresponding evidence mapping.

[0013] The present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and capable of running on the processor, wherein the processor executes the computer program to implement the method described above.

[0014] The present invention also provides a computer-readable storage medium having a computer program stored thereon, which, when executed by a processor, implements the above-described method.

[0015] The present invention also provides a computer program product, including a computer program that, when executed by a processor, implements the above-described method.

[0016] Compared with the prior art, the beneficial effects of the present invention are as follows: Compared to fixed corpus evaluation, random fuzz testing, or matching only by product name, this invention generates probes by reversing the difference state edges of the candidate word segmentation mechanism. This establishes a traceable correspondence between each black-box call and the normalization, boundary segmentation, sub-word merging, special lexical, or template processing behaviors to be distinguished. This invention executes probes within the serialized envelope of real client messages and retains server-side hidden fields as candidate hypotheses, reducing erroneous attributions caused by inconsistencies between the envelope and the raw text. By dynamically deriving equivalence classes from response signatures and worst-case partition selection, it can reduce interface calls and corresponding network transmissions that do not provide distinguishing information under the condition that candidates are separable. Through unknown candidate detection and proof-based stopping, it prevents erroneous unique configuration conclusions when the candidate library is incomplete, interface observations are insufficient, or the budget is exhausted. By changing only a minimal probe pair of a controlled element, it maps output differences to the earliest distinguishable processing node or node range, rather than simply giving results showing a difference in the total number of lexical units. Attached Figure Description

[0017] To more clearly illustrate the specific embodiments of the present invention or the technical solutions in the prior art, the drawings used in the description of the specific embodiments or the prior art will be briefly introduced below. Obviously, the drawings described below are some embodiments of the present invention. For those skilled in the art, other drawings can be obtained from these drawings without creative effort.

[0018] Figure 1 This is a diagram showing the overall structure of the black-box word segmenter adaptive boundary probe and word difference localization system of the present invention.

[0019] Figure 2 A schematic diagram of the structure of the lexicalization state graph for the candidate word segmenter mechanism.

[0020] Figure 3 A flowchart for generating triggering contexts and candidate boundary probes from candidate difference state edges.

[0021] Figure 4 A schematic diagram of the structure for embedding the client's real message serialization envelope into the candidate boundary probe.

[0022] Figure 5 The flowchart describes the process of constructing a response signature, predicting response partitions, and updating equivalence classes for candidate behaviors.

[0023] Figure 6 A flowchart for detecting unknown candidates and stopping the candidate library expansion or limitation.

[0024] Figure 7 The flowchart illustrates the construction of the minimum distinguishing probe pair and the location of the earliest distinguishable processing node.

[0025] Figure 8This is a flowchart showing the output of four types of verifiable stopping conditions and evidence mapping. Detailed Implementation

[0026] The technical solution of the present invention will now be clearly and completely described with reference to the accompanying drawings. Obviously, the described embodiments are only some, not all, of the embodiments of the present invention. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0027] The present invention will be further explained below with reference to specific embodiments.

[0028] like Figure 1 As shown, this embodiment of the invention provides an adaptive boundary probe and lexical difference localization system for a black-box word segmenter, including a candidate modeling module 110, an equivalence class derivation module 120, a probe and envelope module 130, a probe selection module 140, a response validity and update module 150, a difference localization module 160, a stop and evidence module 170, and a data storage area.

[0029] The candidate modeling module 110 is used to acquire multiple candidate segmenter configurations and limited probe domains, construct a mechanism-specific lexicalized state graph, and establish a prediction response signature function. This module reads candidate vocabularies, rules, or executable reference implementations from local storage or an authorized configuration interface, and outputs the candidate state graph. Candidate segmenter configurations can come from publicly available versions, local reference implementations, vendor-published specifications, historical replayable versions, or authorized configurations. Each candidate configuration includes at least a candidate identifier, mechanism type, observable configuration summary, state graph identifier, and predictor version. The candidate identifier is used only for indexing and does not substitute identical product names for identical behavior.

[0030] The equivalence class derivation module 120 is used to dynamically derive candidate behavior equivalence classes based on the relationship of identical predicted response signatures for executed probes configured by the candidate word segmenter. This module reads the state graph prediction results and historical probe records, and outputs the current behavior equivalence class.

[0031] The probe and envelope module 130 is used to generate candidate boundary probes from candidate state edges that enable at least two candidate segmenter configurations in the same current candidate behavior equivalence class to produce different predicted response signatures, and to embed the candidate boundary probes into the client message serialization envelope. This module outputs the raw byte probe and the target state edge, reads the actual template and serialization configuration from the client SDK, and outputs the request payload bytes to be sent.

[0032] The probe selection module 140 is used to determine the target probe from the candidate boundary probes. This module calculates the partition and selects the probe.

[0033] The response validity and update module 150 is used to call the black-box word segmenter and verify the validity of the returned response. It forms a measured response signature from the valid response, compares the measured response signature with the predicted responses of the candidates, and updates the candidate word segmenter configuration and candidate behavior equivalence classes or generates unknown candidate states. This module calls the black-box service through a network interface and converts the returned fields into signatures to update candidates or generate unknown evidence.

[0034] The differential localization module 160 is used to generate minimum distinguishing probe pairs and determine the earliest distinguishable processing node based on the response differences.

[0035] The Stop and Evidence module 170 is used to stop the process when preset stop conditions are met and output the corresponding evidence mapping. This module writes the stop certificate and the evidence mapping.

[0036] The data storage area preserves the original values ​​and integrity summaries of each data object.

[0037] The aforementioned modules can be implemented using program instructions within the same processor, or they can be distributed across the probe generation server, invocation agent, and evidence storage server. If implemented distributedly, the modules exchange messages containing version and integrity verification fields; the encoded version used for response signature calculation must be saved with the evidence record.

[0038] The above is a static description of the system's various components and their functions. The following section combines... Figures 1 to 8 The general execution flow of the present invention will be described in detail, and each step is executed by the corresponding modules mentioned above.

[0039] This invention provides an adaptive boundary probe and lexical difference localization method for a black-box word segmenter, executed by at least one processor, including: Step S201, Candidate Modeling and State Graph Construction. Load the candidate configuration set, observation capabilities, and limited probe domains. For each candidate word segmenter, configure and construct a mechanism-specific lexicalized state graph and verify the integrity of the state edge fields. Candidates that cannot be graphed are not silently deleted but marked as candidates with incomplete models.

[0040] Each candidate configuration is converted into a lexicalized state graph, StateGraph=(V,E). A StateNode records at least the node identifier, processing type, input position or partial symbol sequence, mechanism state, and predecessor node; a State EdgeRecord records at least the edge identifier, preceding state, triggering context, output transformation, successor state, and processing node label. The TriggerContext includes the preceding byte conditions that make the state edge reachable, the target byte conditions, the following constraints, message role or tool pattern constraints, and the applicable observation level.

[0041] The state diagram does not employ a fixed six-layer order, but instead represents message templates or nodes for serialization, normalization, pre-segmentation, sub-word operations, lexicon lookup, and special lexical processing based on the actual processing topology of the candidate implementation. For cases involving branching, backtracking, or repeated execution within the same candidate, state edges record branch conditions, backtracking targets, and execution count constraints. The candidate lexicalization state diagram in this invention is formed by predictions of available candidate configurations or candidate reference implementations, used to generate boundary probes and predict response partitions; it does not imply that the internal structure of the target black box has been directly observed.

[0042] The specific structure of the state diagram for different word segmentation mechanisms is as follows: For the BPE mechanism, the state can consist of the current symbol unit sequence, adjacent candidate pairs, merge ranking, and merge already performed; state edges represent the merge action for the selected ranking. For the WordPiece mechanism, the state can consist of the vocabulary prefix tree node, current position, longest matching candidate, continuation prefix rule, and unknown token backoff state; state edges represent a longest matching or backoff action. For the Unigram mechanism, the state can consist of the input position, candidate token arcs, cost or log probability, dynamic programming cumulative value, and tie-breaking rule; state edges represent candidate arc selection and optimal path transition.

[0043] Configure the candidate word segmenter to predict the response signature function of the probe.

[0044] For candidate configuration c, probe p, and observation capability o, the predicted response signature can be expressed as: Sig(c,p,o)=Encode_o(tokenCount, tokenIdsDigest, byteSpansDigest, specialPositions, scoreVector).

[0045] Specifically, `Encode_o` encodes at least the total number of tokens, and, where observation capability `o` allows, further encodes at least one of the following: ordered token identifiers, UTF-8 byte boundaries, special token positions, probability values, or logarithmic probability values, encoded in the canonical order of field identifiers, lengths, and values. HTTP status codes, ordinary business fields, or error pages cannot constitute the tokenized response signature required for candidate updates on their own. A digest can be calculated for long token identifier sequences or byte boundary sequences, but the original sequence should be preserved in the local reference record. UTF-8 byte boundaries are recorded as half-open intervals [startByte, endByte), preserving a lossless mapping between characters, Unicode code points, and bytes; irreversible normalization uses a mapping from the source span set to the target span, without forging a one-to-one correspondence.

[0046] Step S202, Equivalence Class Derivation. Based on the equivalence relation that the predicted response signatures generated by each candidate word segmenter for the executed probe are the same, candidate behavior equivalence classes are dynamically derived.

[0047] Let the set of executed probes be P_t. In round t, the equivalence relation between candidates c_i and c_j is defined as follows: for each probe in P_t, the predicted response signatures of the two candidates are identical. The candidate behavior equivalence class Class_t is dynamically derived from this relation, rather than being pre-fixed based on product name, version label, or manual label. The equivalence relation is recalculated after adding new tested probes, ensuring that the equivalence class is either maintained or subdivided; the monotonic relation is not forced to be maintained when unknown candidates appear.

[0048] The initial probe can be empty or include fixed seed bytes that do not contain business-sensitive content; the fixed seed does not replace the generation of subsequent state edges.

[0049] Step S203: Generation and envelope embedding of state-edge-driven candidate boundary probes. For behavioral equivalence classes that still contain at least two candidates, the system searches for difference edges in the candidate state graph that satisfy the following conditions: their triggering context is satisfied within the defined probe domain, and at least two candidates in the behavioral equivalence class have different predictions for the response signatures after triggering. A common predecessor is searched backwards from this difference edge to find the shortest preceding context that makes the target state edge reachable; the target byte sequence and the shortest following context that ensures the difference propagates to the observable response are connected to form the original byte-level candidate probe.

[0050] A ProbeRecord includes at least the probe identifier, raw bytes, character encoding, target edge set, triggering context, predicted response partition, message serialization envelope, call cost, and generation rules. Natural language corpora can be used for business representativeness verification, but state edge probes are derived directly from differences in candidate mechanisms and are not randomly sampled from the natural language corpus.

[0051] Optional target edges include Unicode combined characters, compatible decomposition, case folding, whitespace or zero-width character processing edges, script, number, unit, punctuation and code boundaries for regular pre-segmentation, BPE merge ranking edges, WordPiece longest match or unknown backoff edges, Unigram candidate arcs, special lexical allow, prohibit or escape edges, and role, tool call and JSON serialization template edges.

[0052] Each candidate boundary probe is embedded into the client message serialization envelope. The message serialization envelope (EnvelopeRecord) records at least the client request payload bytes, local template output, content type, character encoding, field order, escape rules, role sequence, tool mode, and template version. For candidate probes compared in the same round, the envelope control fields remain consistent, except for the probe carrying fields. The actual request body bytes are frozen and an integrity digest is calculated before transmission; the text displayed on the interface cannot replace the fact of these bytes.

[0053] Server-side hidden templates, route tags, hidden system messages, or hidden special terms are not incorporated into the client's real message serialization envelope. If the observation results support a certain server-derived input, only a ServerDerivedHypothesis is established, recording the hypothesis identifier, candidate fields, supporting probes, contrary evidence, and observable constraints; the hypothesis still participates in candidate inference but is not marked as a directly observed fact.

[0054] Step S204, Probe Selection. The target probe is determined from the candidate boundary probes. For each candidate probe p, a partition Π(p) = {C_1,...,C_k} is formed based on the candidate predicted response signature. Independent target selection first calculates the number of candidates for each partition and selects the candidate probe that minimizes the maximum remaining candidate count max_j||C_j||. In case of a tie, disambiguation is performed sequentially based on call cost, number of transmitted bytes, expected response delay, probe byte length, and probe identifier specification byte order.

[0055] When using weighted selection and equivalence class diameter, the current number of candidates is denoted as M. Candidate weights are derived from a prior distribution with version identifiers; without this prior, each candidate weight defaults to 1 / M, and all weights are then normalized to a sum of 1. Let the set of probes be P*={p_1,...,p_K} and K>0, then the distance between two candidates c_a and c_b is defined as: d(c_a,c_b)=K^{-1}Σ_{k=1}^{K} I[Sig(c_a,p_k,o)≠Sig(c_b,p_k,o)].

[0056] The diameter of the equivalence class of partition C_j is defined as the maximum distance between any two candidates within it; the maximum equivalence class diameter of the probe is the maximum diameter of each predicted response partition. The selector determines the target probe in lexicographical order of minimizing the maximum number of remaining candidates, then minimizing the maximum partition candidate weights, then minimizing the maximum equivalence class diameter, and finally minimizing the call cost. The prior version, K, signature matrix, and normalized weights are written into the selection record, thus enabling repeatable selection results for the same input record.

[0057] Only when the maximum partition size in each round is no greater than half the current candidate size and the candidates are divisible can it be deduced that the number of rounds of calls does not exceed ⌈log2 M⌉; this conclusion is a conditional theoretical upper bound and is not considered as the result of completed experiments.

[0058] Step S205: Black-box call, response comparison, and candidate update. The target probe is used to call the black-box word segmenter to obtain the interface return. The call record (CallRecord) includes at least the request idempotency identifier, probe identifier, request payload byte digest, sending time, original interface return text, response structure version, response fields, network status, retry number, and observation capability. Before candidate update, it is verified that the return structure is complete, the request idempotency identifier is consistent with the target probe, and the network status is valid. Network failures, rate limiting, timeouts, error pages, truncated responses, or unparseable returns only write abnormal evidence and trigger controlled retries; they do not form unknown candidate evidence or change the candidate set.

[0059] The system generates a measured response signature from valid responses. For deterministic interfaces, the system constructs a measured response signature (Sig_obs) from a valid return. For interfaces marked as non-deterministic, the system repeatedly calls the system under the same frozen request bytes and idempotent association rules, generating a measured response distribution signature based on response signature frequency, sample size, and confidence interval. Each candidate configuration generates a predicted response distribution signature under the same repeated call conditions. Only after a preset repeated acknowledgment condition is met is the measured response signature or measured response distribution signature compared with the candidate predicted partition.

[0060] The measured response signature is compared with the predicted response partition, or the measured response distribution signature is compared with the predicted response distribution partition. When a valid and confirmed measured response signature matches a predicted response partition, only candidates within that partition are retained, and a new behavioral equivalence class is derived based on the new probe set containing the probes from this round.

[0061] If no corresponding predicted partition is hit, an unknown candidate state is generated. If a valid and confirmed measured response signature is still not equal to any candidate predicted response signature, an Unknown Evidence is established, saving the original probe, envelope, interface return, idempotency flag, network state, duplicate confirmation record, predicted response signature matrix, and first deviation round. The state is set to Unknown Candidate State (UNKNOWN_CANDIDATE). The system can expand the candidate library and re-predict, or output a blocked state (BLOCKED) or limited state (LIMITED); unknown responses must not be forcibly assigned to the closest candidate based on edit distance or numerical distance.

[0062] When the black-box interface only returns the usage field, the measured response signature degenerates into the total number of tokens, and the observation capability is marked as count-only (COUNT_ONLY). In this case, the system outputs the count-level equivalence class and the limited state (LIMITED), and does not presume that the token identifiers or boundaries are the same just because the total number is the same.

[0063] If the network fails, rate is limited, timeout occurs, error page appears, or the response is truncated, maintain the candidate update state and retry according to the backoff strategy or output evidence of abnormal obstruction.

[0064] Step S206: Minimal Discrimination Probe Pairs and Differential Node Location. For candidate behavior equivalence classes that remain distinguishable after the update, generate minimum discrimination probe pairs that differ only in one controlled byte, one controlled byte span, or one message serialization envelope field. The system compares two candidate state paths and finds the first separating state edge of the predicted path. Based on the common predecessor context of this edge, construct a first probe p_a and a second probe p_b, which differ only in one controlled byte, one consecutive byte span, or one message envelope field, while keeping the remaining envelope bytes the same.

[0065] If the measured response difference matches the predicted difference of the target state edge, and the predicted node outputs of the two candidates before the target edge are the same, then the node to which the target edge belongs is marked as the earliest identifiable processing node. If observations limit the exclusion of only one set of nodes, then the output ranges from the common predecessor to the first observable difference node, without fabricating direct observations of internal nodes. If the results are inconsistent, then the context is expanded to the predecessor, another minimal pair is selected, or the unknown candidate is processed.

[0066] The minimum probe pair record DistinctionPair should include at least the common bytes, changed position, values ​​before and after the change, target edge, two candidate predicted paths, measured response signature, localization conclusion, and localization information status.

[0067] Step S207, Stop the determination and evidence output. Check the stopping conditions. Stop when any of the following conditions are met: the candidate segmenter configuration converges to the unique candidate segmenter configuration; the remaining candidates have indistinguishable proof of response signatures within the defined probe domain; the preset call budget is reached; or the unknown candidate state is generated. Output the evidence mapping between the probe, candidate state edge, predicted response partition, measured response signature or measured response distribution signature, and the earliest distinguishable processing node. The unique candidate segmenter configuration conclusion is not output when the preset call budget is reached or the unknown candidate state is generated.

[0068] A StopCertificate record must include at least the stop type, candidate snapshots, executed probes, predicted response signature matrix, qualified probe domain description, budget record, unknown evidence, and integrity summary. Stop types include: 1. Unique Candidate State (UNIQUE_CANDIDATE): Only one candidate segmenter configuration remains in the candidate set; output the candidate segmenter configuration and its evidence.

[0069] 2. Indistinguishable state within the domain (INDOMAIN_EQUIVALENT): The remaining candidate pairs generate the same predicted response signature for all enumerated trigger edges in the limited probe domain, and save the domain integrity proof; output the conclusion that the state is indistinguishable within the domain.

[0070] 3. Budget Exhausted (BUDGET_EXHAUSTED): The maximum number of calls, bytes transferred, latency, or authorized resource budget has been reached; output the LIMITED and Uncovered states edges, and do not output the unique candidate.

[0071] 4. Unknown Candidate State (UNKNOWN_CANDIDATE): The valid and repeatedly confirmed measured response signature still does not hit any prediction partition; output unknown evidence and blocked or limited state, and do not output a unique candidate.

[0072] The probe domain must be computably described, for example, by a given set of Unicode code points, maximum byte length, role combination, tool mode, set of mechanism state edges, and length of the back context; subjective judgments such as "has been sufficiently tested" cannot replace proof of domain integrity.

[0073] If the iteration does not stop, return to step S203 and continue iterating.

[0074] The above general execution flow describes the overall processing framework of this invention in various candidate scenarios. To more clearly demonstrate the specific implementation methods and effects of this invention in different practical application scenarios, several embodiments are illustrated below. Each embodiment addresses the differences in word segmentation mechanisms or interface observation limitations of different types, demonstrating the complete process from probe generation to stopping output.

[0075] Example 1: Unicode Normalized State Edge Recognition The candidate configurations in this embodiment include three types of behavior: not performing Unicode normalization, performing NFC normalization, and performing NFKC normalization. The candidate modeling module 110 establishes normalized state edges for each type and records the trigger predicates for combined characters, pre-assembled characters, and compatible characters. The probe field is limited, for example, to a selected code point set, a UTF-8 length not exceeding a preset value, and the use of only user message roles.

[0076] The probe and envelope module 130 generates candidate probes by reversing the difference edges between "combining sequences into pre-assembled characters" and "folding compatible characters into standard characters". For example, the first probe uses pre-assembled characters, and the second probe uses base characters plus combined additional symbols; the roles, field order, and escaping in the envelopes of both probes are consistent. The probe selection module 140 selects target probes after forming partitions based on the predicted Token ID and UTF-8 boundaries.

[0077] When the black box returns observable lexical identifiers or boundaries, the response validity and update module 150 forms a multi-field response signature; if only the total number is returned, a count-level signature is formed. After the response validity and update module 150 updates the candidate classes, the difference localization module 160 constructs the minimum probe pair that only changes the combination method. If the measured difference is consistent with the normalized edge prediction, the earliest distinguishable node is output as the normalized processing node; if NFC and NFKC cannot be distinguished by count alone, the count-level equivalence class and restricted state are output.

[0078] Example 2: BPE merging and WordPiece rollback difference location This embodiment differs from Embodiment 1 in that it focuses on the differences in Unicode standardization. Instead, it focuses on the mechanism differences between different sub-word algorithms (BPE and WordPiece), as well as the ranking or vocabulary differences between different versions within the same algorithm. At least one BPE candidate and at least one WordPiece candidate are set. The BPE state graph records symbolic units, adjacent merge pairs, and rankings; the WordPiece state graph records prefix tree paths, longest matches, and unknown term backoffs. The system searches for difference edges where the same byte context triggers a merge in the BPE candidate, or triggers a longest match or backoff in the WordPiece candidate.

[0079] The probe and envelope module 130 constructs a probe consisting of a common prefix, a target substring, and the shortest suffix, and embeds it with the same user message envelope. The probe selection module 140 prioritizes probes with smaller maximum prediction partitions. If there are still multiple BPE ranking versions after the first call, the difference localization module 160 constructs the smallest probe pair that only changes one adjacent byte from its first different ranking edge; if there are still multiple WordPiece vocabulary versions after the first call, the difference localization module 160 constructs the smallest probe pair from the longest matching competition position after the shared prefix.

[0080] If the interface returns a Token ID, the response signature can distinguish candidates with the same total number of tokens but different ID sequences; if it returns a byte boundary, it can distinguish changes in the boundary position. The system outputs the target merged edge or the longest matching / backoff node and its corresponding evidence.

[0081] Example 3: Behavior Recognition of Roles and Tools in JSON Message Templates In this embodiment, the candidates use the same raw text sub-word mechanism, but the client-observable templates differ in role prefixes and suffixes, tool definition field order, empty client system messages, and JSON escaping methods. The system generates four types of triggering contexts from the template serialization state edge: single user messages, user messages containing empty client system messages, messages that enable tool mode, and tool messages containing nested JSON parameters.

[0082] The probe and envelope module 130 calls the serialization entry point of the client's actual SDK, or constructs a request based on a frozen local template, saving the actual request payload bytes to be sent. Regarding whether the server injects a hidden template separately, the system only establishes candidate hypotheses. The probe selection module 140 selects probes while maintaining consistency between the role sequence and tool mode; the difference localization module 160 constructs minimal probe pairs that only change the order of one JSON field, one escape character, or one role field.

[0083] When the measured response matches a certain template behavior class, the system outputs the behavior class and observable envelope evidence; when multiple candidate templates are indistinguishable under the current interface field, the system outputs the indistinguishable or restricted state within the domain.

[0084] Example 4: Only returns usage and contains non-deterministic responses Unlike the previous embodiments where the interface could return lexical identifiers or byte boundaries, this embodiment examines the case where the interface's observation capabilities are most limited—only returning the total number of lexical uses. The black-box interface in this embodiment only returns the total number of input lexical uses, and multiple count values ​​may appear for the same request due to server routing or canary releases. The system marks the observation capability as count-only (COUNT_ONLY), establishes an idempotent flag for each target probe, and calls it a preset number of times.

[0085] The response validity and update module 150 constructs a distributed response signature from the frequency distribution of the same probe counts; the equivalence class derivation module 120 derives a coarse-grained equivalence class based on this signature, without using a single random value. If the remaining candidates have the same count distribution within the defined probe domain, the stopping and evidence module 170 generates an indistinguishable (INDOMAIN_EQUIVALENT) stopping certificate; if the budget is exhausted first, a budget exhaustion (BUDGET_EXHAUSTED) stopping certificate is generated. In both cases, uncovered state edges and restricted states are output.

[0086] Example 5: Unknown Candidate Detection and Candidate Expansion All the preceding embodiments assume that the actual behavior of the black box is contained in the candidate library. This embodiment examines the case where the candidate library is incomplete, verifying the effectiveness of the system's unknown candidate detection mechanism. The preset candidate library in this embodiment does not contain a single newly added merging rule actually used by the black box. After the system selects target probes for several known merging ranking differences, the measured TokenID digest or boundary digest does not fall into any prediction partition. The response validity and update module 150 generates an unknown candidate (UNKNOWN_CANDIDATE), saves the request payload, the returned original text, all predicted response signatures, and the first deviation round, and the stop and evidence module 170 stops outputting the unique candidate.

[0087] During the candidate expansion phase, the system generates state edge requirements for supplementary candidates based on the difference between the unknown response and the known prediction. After authorizing the addition of new candidate implementations, the predicted response signature is recalculated. If the new candidate can explain the response, it proceeds to subsequent probe rounds; otherwise, it outputs the blocked state and the minimum deviation probe pair.

[0088] The output of each of the above embodiments is an evidence map, which is a structured data record that is machine-readable, verifiable, and replayable. An exemplary structure of the evidence map is given below.

[0089] Example structure of evidence output: Evidence mapping uses machine-readable records, including at least: case or experiment identifier, candidate library version, limited probe domain version, probe generation rules, target state edge, client request payload digest, prediction partition, API return plaintext digest, measured response signature, snapshots before and after candidate updates, minimum probe pair, location conclusion, stop type, observable limitations, and anomaly and integrity list. Evidence mapping is used for technical review and replay, and does not rely on specific evidence storage technologies.

[0090] Finally, it should be noted that the above embodiments are only used to illustrate the technical solutions of the present invention, and not to limit them; although the present invention has been described in detail with reference to the foregoing embodiments, those skilled in the art should understand that modifications can still be made to the technical solutions described in the foregoing embodiments, or equivalent substitutions can be made to some or all of the technical features; and these modifications or substitutions do not cause the essence of the corresponding technical solutions to deviate from the scope of the technical solutions of the embodiments of the present invention.

Claims

1. A black-box word segmenter adaptive boundary probe and lexical difference localization method, characterized in that, Executed by at least one processor, including: Obtain multiple candidate segmenter configurations and limited probe domains, construct a mechanism-specific lexicalization state graph for each candidate segmenter configuration, and establish a prediction response signature function for the candidate segmenter configuration to the probe; Based on the equivalence relation that the predicted response signatures generated by each candidate word segmenter for the executed probe are the same, the candidate behavior equivalence classes are dynamically derived. Candidate boundary probes are generated from candidate state edges that enable at least two candidate segmenters in the same current candidate behavior equivalence class to produce different predicted response signatures, and each candidate boundary probe is embedded into the client message serialization envelope. Determine the target probe from the candidate boundary probes; The target probe is used to call the black-box word segmenter, obtain the interface return, form a measured response signature from the valid response, and compare the measured response signature with the candidate predicted response to update the candidate word segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state. Generate minimum distinguishing probe pairs, and determine the earliest distinguishable processing node based on the response differences of the minimum distinguishing probe pairs; Stop when the preset stopping conditions are met, and output the corresponding evidence mapping.

2. The method according to claim 1, characterized in that, The state edges of the lexicalized state graph at least record the previous state, triggering context, candidate output, and successor state. The mechanism-specific lexical state graph includes at least one of the following: For a byte-pair encoding word segmenter, the state is composed of the current symbol unit sequence, adjacent candidate pairs, and merge ranking, and the state edges are composed of performing the selected merge and updating the adjacent candidate pairs. For the WordPiece word segmenter, the state is composed of the vocabulary prefix node, the current input position, the longest matching candidate and the unknown word backoff flag, and the state edge is composed of the execution of the longest match or the unknown word backoff. For the Unigram segmenter, the state is composed of input position, candidate lexical arc, candidate cost, cumulative path value and parallel decision marker, and the state edge is composed of selecting candidate lexical arc and updating the optimal path. Generating candidate boundary probes from the candidate state edges includes: generating candidate boundary probes from candidate state edges and their triggering contexts that enable at least two candidate word segmenters in the same current candidate behavior equivalence class to generate different predicted response signatures; specifically, it includes: inversely calculating the shortest preceding byte sequence that makes the target state edge reachable, connecting the target byte sequence that makes the target state edge valid and the shortest following byte sequence that keeps the subsequent response observable. The predicted response signature and the measured response signature include at least the total number of tokens, and when the interface is available, further include at least one tokenized observable among ordered token identifiers, UTF-8 byte boundaries, special token positions, probability values, or logarithmic probability values; the response signature is normalized and encoded according to a predetermined field order, and the source span set, target span, and mapping type are irreversibly normalized and stored. The client message serialization envelope is the client real message serialization envelope. The client real message serialization envelope consists of at least one of the following: client request payload bytes, character encoding, field order, escape rules, role sequence, and tool definition that are observable by the client before transmission. Unobservable server-side hidden fields are not treated as known fields. For server-side hidden templates or hidden special terms, they are only recorded as candidate hypotheses with support probes and observability constraints. Determining the target probe from the candidate boundary probes includes: calculating the predicted response partitions formed by each candidate boundary probe with respect to the current candidate segmenter configuration, and determining the target probe from the candidate boundary probes with minimizing the maximum number of remaining candidates in the predicted response partition as the selection objective; the candidate weights are derived from a prior distribution with version identifiers, and the default weights of the current candidates are equally normalized when the prior distribution does not exist; within the limited probe set, the distance between two candidates is defined as the proportion of probes whose predicted response signatures are different; the equivalence class diameter of a predicted response partition is defined as the maximum distance between any two candidates in that partition; for each candidate boundary probe, the maximum number of remaining candidates, the candidate weight of the maximum partition, and the maximum equivalence class diameter are calculated, and the target probe is determined in lexicographical order of first minimizing the maximum number of remaining candidates, then minimizing the candidate weight of the maximum partition, and then minimizing the maximum equivalence class diameter.

3. The method according to claim 1, characterized in that, The measured response signature is compared with the candidate predicted response to update the candidate segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state, specifically including: For deterministic interfaces, a measured response signature is generated from valid responses; for black-box interfaces marked as nondeterministic, the same target probe is executed according to preset repeated acknowledgment conditions to generate a measured response distribution signature. Compare the measured response signature with the predicted response partition, or compare the measured response distribution signature with the predicted response distribution partition; When a corresponding prediction partition is hit, the candidate segmenter configuration and candidate behavior equivalence class are updated. When no corresponding prediction partition is hit, an unknown candidate state is generated. The process of generating the unknown candidate state includes: confirming that the interface returns a response that conforms to a predetermined response structure, that the request idempotent identifier is consistent with the target probe, and that the applicable duplicate confirmation conditions are met; after meeting the above conditions, determining that the measured response signature or measured response distribution signature is different from all candidate predicted response signatures or predicted response distribution signatures; saving the original bytes, envelope, interface return text, and time information of the target probe; and ensuring that the unknown response is not forcibly merged into the nearest predicted response partition.

4. The method according to claim 1, characterized in that, Generating the minimum distinguishing probe pair includes: reversing the common pre-context from the first separated state edge of the two candidate state paths, constructing a first probe and a second probe that are identical except for a controlled byte, a controlled byte span, or a message serialization envelope field; when the measured response signature difference between the first probe and the second probe is consistent with the predicted difference of the first separated state edge, the processing node corresponding to that state edge is determined as the earliest distinguishable processing node.

5. The method according to claim 3, characterized in that, When a black-box call results in network failure, rate limiting, timeout, page error, or truncated response, an idempotent call identifier is established for the same probe, and the state before the candidate update is maintained. The call is repeated according to the backoff strategy. Network failure, rate limiting, timeout, page error, or truncated response are only recorded as abnormal evidence without updating the candidate set or generating an unknown candidate state. When the same probe generates multiple valid responses, the measured response distribution signature is formed based on the frequency and confidence interval of the response signature, and compared with the predicted response distribution signature formed by each candidate configuration under the same repeated call conditions.

6. The method according to claim 1, characterized in that, The preset stopping conditions include at least one of the following: the candidate segmenter configuration converges to the unique candidate segmenter configuration, the remaining candidates have an indistinguishable proof of response signature within the defined probe domain, the preset call budget is reached, and the unknown candidate state is generated. When the black-box interface only returns the total number of tokens used, the total number of tokens used is used to construct a count-level response signature, and the count-level candidate behavior equivalence class and observability constraint are output; the indistinguishability proof includes the generation rules of the limited probe domain, the enumerated set of probes or state edges, the response signature matrix of the remaining candidates, and the integrity digest.

7. A black-box word segmenter adaptive boundary probe and lexical difference localization system, performing the method as described in any one of claims 1 to 6, characterized in that, include: The candidate modeling module is used to obtain multiple candidate word segmenter configurations and limit probe domains, construct a mechanism-specific lexicalization state graph, and establish a predictive response signature function; The equivalence class derivation module is used to dynamically derive candidate behavior equivalence classes based on the relationship between the predicted response signatures of the executed probes and the candidate word segmenters. The probe and envelope module is used to generate candidate boundary probes from candidate state edges that enable at least two candidate segmenters in the same current candidate behavior equivalence class to generate different predicted response signatures, and to embed the candidate boundary probes into the client message serialization envelope. A probe selection module is used to determine the target probe from the candidate boundary probes; The response validity and update module is used to call the black-box word segmenter and verify the validity of the return, form a measured response signature from the valid response, and compare the measured response signature with the candidate predicted response to update the candidate word segmenter configuration and candidate behavior equivalence class or generate an unknown candidate state. The differential localization module is used to generate minimum distinguishing probe pairs and determine the earliest distinguishable processing node based on response differences; It also includes a stop and evidence module, which is used to stop when a preset stop condition is met and output the corresponding evidence mapping.

8. An electronic device comprising a memory, a processor, and a computer program stored in the memory and capable of running on the processor, characterized in that, When the processor executes the computer program, it implements the method of any one of claims 1 to 7.

9. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

10. A computer program product, comprising a computer program, characterized in that, When the computer program is executed by a processor, it implements the method of any one of claims 1 to 7.

Images