Semantic-aware black-box large language model training data auditing method and system

Abstract

The present application relates to the technical field of training data auditing, in particular to a semantic-aware black-box large language model training data auditing method and system, comprising the following steps: returning content based on a text generation interface, analyzing word element and candidate content multi-round sampling distribution, determining convergence index, calculating semantic path and weight of real content and candidate content, analyzing difference with reference content, and obtaining attribution adaptation determination result. Through multi-round sampling behavior fluctuation tracking, section feature dynamic extraction, stability change judgment and sequence level weight aggregation, the present application realizes fine separation of training data attribution signal, constructs a multi-level judgment system for complex expression and diverse output, optimizes discrimination accuracy through tension comparison signal group and attribution adaptation determination mechanism, forms a sensitive and adaptive training data member auditing strategy, effectively prevents risk hidden danger omission and strengthens model data security boundary control.

Description

TECHNICAL FIELD

[0001] The present application relates to the technical field of training data auditing, in particular to a semantic-aware black-box large language model training data auditing method and system. BACKGROUND

[0002] The technical field of training data auditing involves comprehensive quality detection and risk assessment of data sets used for machine learning model training, ensuring that data sources are legal, content is accurate, labels are consistent, and structure is reasonable. Training data auditing can effectively identify potential security risks, ethical issues, or potential biases in the training data set, and provide reliable and controllable data foundation for model training. Among them, the traditional black-box large language model training data auditing method refers to a method of evaluating the quality and source of training data by combining semantic understanding mechanism based on the premise of not accessing the internal parameters and structure of the model, only through the input and output results of the model. It usually adopts the method of constructing specific input samples, analyzing the semantic relevance of the model output response, comparing the word vector distribution, judging the context consistency, etc., to infer the training corpus that the model may have accepted, and further audit the potential risks of the training data.

[0003] The existing method relies on the surface information of input and output to make sample judgments, and it is difficult to identify when the model response distribution is diverse or the context expression is complex. The mechanism of static sampling and point-to-point semantic comparison cannot capture the dynamic characteristics of the model generation behavior, and it lacks the perception of the distribution changes of the candidate content in the multi-round sampling process, lacks the sensitive positioning of the information change section, and is easy to cause the misjudgment of the membership of the training data. Risk factors are difficult to expose in time, and the auditing process is easily affected by the uncertainty of the model and the diversity of expression, resulting in single judgment standard and blind area in data security protection. SUMMARY

[0004] In view of the deficiencies of the prior art, the technical problem to be solved by the present application is to provide a semantic-aware black-box large language model training data auditing method and system.

[0005] In order to achieve the above purpose, the present application adopts the following technical scheme, a semantic-aware black-box large language model training data auditing method, comprising the following steps:

[0006] S1: returning content based on a text generation interface, obtaining candidate content through multi-round sampling, comparing the appearance frequency of each word element content, identifying the section with frequency fluctuation in continuous sampling, determining the fluctuation trend, and obtaining the sequence distribution fluctuation feature;

[0007] S2: based on the sequence distribution fluctuation feature, filtering the word element with frequency fluctuation trend to be stable as the convergence point, adjusting the convergence standard for high-frequency sampling position, re-determining the convergence condition of the change section, and obtaining the convergence index group;

[0008] S3: based on the convergence index set, calculate the semantic path between the real content and the candidate content, analyze the semantic span and the order of the intermediate content in the path, integrate the path span and order information to obtain a semantic evidence weight set;

[0009] S4: based on the semantic evidence weight set, filter the content position with information change, judge the content change range before and after the content position point, pair the semantic support change and information fluctuation in the window one by one to obtain a tension comparison signal group;

[0010] S5: based on the tension comparison signal group, adjust the pairing mode with the non-member reference content, analyze the difference between the current sample and the reference content, optimize the decision boundary according to the matching degree, and judge the distribution of confusion and semantic support to obtain a belonging adaptation judgment result.

[0011] The application improves that the sequence distribution fluctuation feature includes a distribution stability label, a fluctuation critical point and a trend identification parameter, the convergence index set includes a position identifier, a stability level and a standard adjustment factor, the semantic evidence weight set includes a weight distribution factor, a coverage coefficient and a path mapping index, the tension comparison signal group includes a signal difference amount, a section change coefficient and a pairing result label, and the belonging adaptation judgment result includes a belonging classification, a level label and a probability interval.

[0012] The application improves that the sequence distribution fluctuation feature is obtained by the following steps:

[0013] S111: based on the text generation interface returning content, analyze the context content corresponding to each word position in the target sequence, compare the appearance frequency of each word in the sampling result after multiple rounds of sampling, calculate the number difference of each candidate word in each sampling round, judge the frequency distribution change to obtain a sampling frequency distribution sequence;

[0014] S112: based on the sampling frequency distribution sequence, compare the frequency change data of each word position, identify the continuous section with a change amplitude in multiple samplings, judge the change trend and characteristics of the section to obtain a fluctuation section label set;

[0015] S113: based on the fluctuation section label set, calculate the candidate word frequency distribution corresponding to each section, analyze the change trend and order of each position in the section, judge the distribution change type in each section to obtain the sequence distribution fluctuation feature.

[0016] The application improves that the convergence index set is obtained by the following steps:

[0017] S211: Based on the sequence distribution fluctuation characteristics, the frequency distribution change of each word element in the context window is analyzed, the distribution sequence of each word element in the candidate content in multiple rounds of sampling is judged, the position where the change gradually slows down and the difference gradually reduces is identified by comparing the distribution convergence of each sampling result, and a convergence position index set is obtained.

[0018] S212: Based on the convergence position index set, the candidate word element frequency distribution corresponding to each position is compared with the aggregation characteristics of the adjacent position, the distribution difference of the internal aggregation density of the section is judged, the section in the sampling frequency aggregation feature set is identified, and an aggregation feature section group is obtained.

[0019] S213: Based on the aggregation feature section group, the candidate word element frequency distribution structure of the corresponding section is calculated, the convergence performance and sequence attribution trend of each position in the section are analyzed, the distribution classification of the aggregation section is judged, and a convergence index group is obtained.

[0020] The application improves that the acquisition step of the semantic evidence weight set is specifically:

[0021] S311: Based on the convergence index group, the corresponding relationship between each candidate word element and the real word element in the semantic space is analyzed, the path structure of the semantic link between the candidate content and the real content is compared, the word element order of each jump link in the path is judged, and a semantic link structure set is obtained.

[0022] S312: Based on the semantic link structure set, the semantic span of each path is compared, the semantic distance from the starting point to the ending point of each path and the distribution change of the jump link content are calculated, the parameter relationship between the span and the jump link characteristics is analyzed, and semantic span mapping data is obtained.

[0023] S313: Based on the semantic span mapping data, the ordering position of the intermediate word element in each path in the sampling sequence is judged, the distribution law of the ordering data and the path span characteristics is analyzed, the fusion mode of the ordering parameter and the span parameter is optimized, and a semantic evidence weight set is obtained.

[0024] The application improves that the acquisition step of the tension comparison signal group is specifically:

[0025] S411: Based on the semantic evidence weight set, the information distribution characteristics of each word element in the sequence are analyzed, the semantic support parameter change trend is judged, the fluctuation performance of the information intensity at each word element position is compared, the content point showing information mutation or consistency reduction in the distribution is identified, and a variation feature index is obtained.

[0026] S412: Based on the variation feature index, the semantic distribution range of the target position and the adjacent word element is judged, the continuity of the semantic structure before and after each position is compared, the offset performance of the context distribution is analyzed, the information fluctuation and the segment of distribution difference are identified, the position is divided according to the interval, and the segment fluctuation window is obtained.

[0027] S413: Based on the segment fluctuation window, the semantic support data and the information fluctuation parameter are called, the change direction and the structure feature of two types of signals in each window are analyzed, the pairing difference between signals in the same group is judged, and the tension comparison signal group is obtained.

[0028] The application improves that the acquisition step of the home adaptation judgment result is specifically:

[0029] S511: Based on the tension comparison signal group, the pairing structure of non-member reference content in the current sample and the training sample is analyzed, the distribution performance of each paired signal in the difference sample is compared, the influence of the pairing mode on the matching result between signals is judged, and the signal pairing configuration is obtained.

[0030] S512: Based on the signal pairing configuration, the matching degree between the current sample signal and the non-member reference content signal is judged, the belonging situation of each type of signal matching in the interval boundary is compared, the difference between the distribution characteristics is analyzed, and the sample classification parameter group is obtained.

[0031] S513: Based on the sample classification parameter group, the sample confusion parameter and the semantic support parameter distribution are calculated, the distribution characteristics of signal types in the full sequence range are analyzed, the belonging type and the probability interval are judged, and the home adaptation judgment result is obtained.

[0032] The application improves that the fluctuation trend refers to whether the sampling frequency of the segment presents a stable, rising, falling or dramatic change state by comparing the distribution of each round of sampling results, and the convergence condition of the change segment refers to whether the sampling distribution of all word elements in the target segment reaches a stable state as a whole or locally.

[0033] A semantic-aware black-box large language model training data auditing system, the system comprises:

[0034] The fluctuation feature analysis module returns the content based on the text generation interface, obtains the candidate content through multiple rounds of sampling, compares the frequency of each word element content, identifies the segment of frequency fluctuation in continuous sampling, judges the fluctuation trend, and obtains the sequence distribution fluctuation feature;

[0035] The convergence judgment module filters the word element with fluctuating frequency tending to be stable as a convergence point based on the sequence distribution fluctuation feature, adjusts the convergence standard for high-frequency sampling position, rejudges the convergence condition of the change segment, and obtains the convergence index group.

[0036] The semantic weight calculation module calculates a semantic path between the real content and the candidate content based on the convergence index group, analyzes a semantic span and an order of path intermediate content, integrates path span and order information, and obtains a semantic evidence weight set;

[0037] The tension pairing module filters content positions with information changes, judges content change ranges before and after the content position point, pairs semantic support changes and information fluctuations in the window one by one based on the semantic evidence weight set, and obtains a tension comparison signal group.

[0038] The attribution analysis module adjusts a pairing mode with non-member reference content, analyzes differences between the current sample and the reference content, optimizes a decision boundary according to a matching degree, and judges distribution of confusion and semantic support, and obtains an attribution adaptation decision result.

[0039] Compared with the prior art, the present application has the beneficial effects that in the present application, through multi-round sampling behavior fluctuation tracking, section feature dynamic extraction, stability change judgment and sequence level weight aggregation, fine separation of training data attribution signals is realized, distributed fluctuations, convergence and semantic evidence mapping modes are fully utilized to improve recognition ability, a multi-level decision system is constructed for complex expressions and various outputs, tension comparison signal groups and attribution adaptation decision mechanisms are used to optimize discrimination accuracy, a sensitive and adaptive training data member audit strategy is formed, risk hidden dangers are effectively prevented from being missed, and model data security boundary control is strengthened. BRIEF DESCRIPTION OF DRAWINGS

[0040] Figure 1 The main step flowchart of the present application is shown in the figure.

[0041] Figure 2 The sequence distribution fluctuation feature acquisition flowchart in the present application is shown in the figure.

[0042] Figure 3 The convergence index group acquisition flowchart in the present application is shown in the figure.

[0043] Figure 4 The semantic evidence weight set acquisition flowchart in the present application is shown in the figure.

[0044] Figure 5 The tension comparison signal group acquisition flowchart in the present application is shown in the figure.

[0045] Figure 6 The attribution adaptation decision result acquisition flowchart in the present application is shown in the figure.

[0046] Figure 7 The training data audit flowchart in the present application is shown in the figure. DETAILED DESCRIPTION

[0047] Specific embodiments of the present invention are given below. These specific embodiments are only used to further illustrate the present invention in detail and do not limit the scope of protection of the present invention.

[0048] Example

[0049] Please see Figure 1 This invention provides a method for auditing training data of a semantically aware black-box large language model, comprising the following steps:

[0050] S1: Based on the content returned by the text generation interface, analyze the position of each word in the target sequence, sample all subsequent candidate content at each position in multiple rounds, compare the frequency of each word content in the sampling, identify the content segments with large fluctuations in frequency during continuous sampling, combine the sampling distribution of candidate content, determine the fluctuation trend, and obtain the sequence distribution fluctuation characteristics.

[0051] S2: Based on the sequence distribution fluctuation characteristics, determine the changing trend of each word position in the context, identify words whose frequency fluctuation range tends to be stable in continuous sampling as convergence points, adjust the convergence criteria for the sampling positions where the frequency of the candidate word sample sequence is higher than that of other content, re-discriminate the convergence of the change segment, and obtain the convergence index group.

[0052] S3: Based on the convergence index group, calculate the semantic path between the real content and the candidate content, analyze the semantic span between the candidate content and the real content, determine the order of the path intermediary content in all sampled content, and integrate the path span and order information to obtain the semantic evidence weight set.

[0053] S4: Based on the semantic evidence weight set, filter the content positions with information changes in the word sequence, determine the range of changes of adjacent content before and after the content position, and combine the comparison parameters of information distribution tension to pair the semantic support changes and information fluctuations within the window. Process the changes one by one using a calculation method to obtain the tension comparison signal group.

[0054] S5: Based on the tension contrast signal group, adjust the pairing method with non-member reference content in the training samples, analyze the difference between the current sample and the reference content, optimize the judgment boundary if the matching degree falls to the lower limit of the reference interval, and move the boundary up if the matching degree is at the upper limit. Also, determine the distribution of confusion and semantic support in the current sample to obtain the attribution and fitting judgment result.

[0055] The sequence distribution fluctuation characteristics include distribution stability labels, fluctuation critical points, and trend identification parameters. The convergence index set includes location identifiers, stability grades, and standard adjustment factors. The semantic evidence weight set includes weight distribution factors, coverage coefficients, and path mapping indicators. The tension comparison signal set includes signal difference, segment change coefficients, and pairing result labels. The attribution and fitting determination results include attribution classification, grade labels, and probability intervals.

[0056] Specifically, E1: for a given text sequence to be audited, at each token position, all tokens before it are taken as context input, and the text generation interface of the target black-box large language model is called. By performing an adaptive Monte Carlo sampling process, a set of candidate tokens for the next token can be obtained, and based on this, an empirical probability distribution reflecting the model's prediction tendency can be constructed.

[0057] As shown in Figure 7 When processing the target word "USA", first extract its text prefix "The 1984 Summer Olympics were held in" as context. Then, enter the "adaptive Monte Carlo sampling" module, which is the core task of this module to empirically estimate which tokens are most likely to be generated by the model after the context by calling the text generation interface of the target model multiple times;

[0058] The key is that the number of samplings K i is adaptively determined rather than a fixed value, ensuring that computing resources are used efficiently. First, perform an initial number of samplings, then calculate the Shannon entropy of the empirical probability distribution formed by all current sampling results. Shannon entropy is an index in information theory to measure uncertainty. A higher entropy value means that the model's prediction at this position is very scattered and uncertain, which is the key position that needs the most attention in auditing, because the model's prediction on non-training data is usually more divergent. If the entropy value is higher than the preset stability threshold, increase the number of samplings and recalculate the entropy value until the entropy value is lower than the stability threshold or reaches the upper limit of sampling. The formula is as follows:

[0059]

[0060] Where, represents the final calculation value of Shannon entropy, p i is the empirical probability distribution of the i-th position in the text sequence to be audited, is the set of all unique candidate tokens obtained by sampling the i-th position, and the summation variable c represents a specific unique candidate token in the set. p i (c) represents the specific probability value of candidate token c under the empirical probability distribution;

[0061] After the adaptive sampling process is completed, the empirical probability distribution p i is obtained by counting the frequency of each unique candidate token c, and the calculation formula is as follows:

[0062]

[0063] Where, p i(c) is the probability of the candidate token c, count(c) is the total number of times the token appears in K i total number of times the token appears in K i is the final total number of samples, refer to Figure 7 For example, after sampling, the model can generate "America" with a probability of P = 2 / 5, "Canada" with a probability of P = 2 / 5, and "USA" with a probability of P = 1 / 5.

[0064] E2: Calculate a "semantic-aware density" that integrates semantic information for the true token at the current position, while also calculating an "information weight" determined by the Shannon entropy of the empirical probability distribution. The weight is proportional to the uncertainty of the model's prediction at this position, that is, the more uncertain the prediction, the higher the weight.

[0065] One, calculate the semantic-aware density, refer to Figure 7 the "semantic-aware density calculation" module in the present scheme, instead of simply using the empirical probability of 1 / 5 of "USA" as a signal, a semantic aggregation mechanism is used to reassemble the probability evidence scattered in multiple semantically related candidate tokens into the true token t i , the formula is defined as:

[0066]

[0067] wherein, is the semantic-aware density of the true token t i given the previous context , and is the set of all unique candidate tokens obtained at position i, p i (c) is the empirical probability of the candidate token c at this position, sim(c, t i ) represents the binary semantic association weight from the candidate token c to the true token t i , and ε is a preset minimum positive number to ensure the numerical stability of subsequent logarithmic operations.

[0068] The calculation of the semantic association weight sim(c, t i ) is determined by a pre-trained natural language inference (NLI) model, which aims to accurately determine whether the semantic of the candidate token c can derive the semantic of the true token t i . Specifically, the candidate token c is used as the input premise of the NLI model, and the true token t iAs an input hypothesis, the NLI model classifies the pair of (premise, hypothesis) into one of the following relationships: "entailment", "contradiction", or "neutral", with a semantic correlation weight sim(c, t i ) that is a hard label based on the model's decision, defined as follows:

[0069]

[0070] This binary weight function ensures that only the candidate tokens that constitute strict positive semantic evidence (i.e., entailment) are assigned a weight of 1 and their probabilities are aggregated; all other irrelevant or semantically contradictory candidate tokens are assigned a weight of 0, thus completely eliminating noise interference.

[0071] For example, for the candidate token "America" and the true token "USA", the NLI model determines that it constitutes an "entailment" relationship, so , on the contrary, for the candidate token "Canada", the model determines that the relationship is "neutral" or "contradiction", so , in this way, as Figure 7 shows, the final semantic perception density adds the probability evidence of "USA" and "America" (1 / 5 + 2 / 5 = 3 / 5), obtaining a much stronger and more accurate memory signal than the original lexical probability.

[0072] Second, calculate the information weight. A key design in this embodiment is to highlight the position with the most information about membership by calculating the information weight w i , which is determined by the model's uncertainty in predicting this position and quantified by Shannon entropy H(p i ). The principle is that the strongest memory evidence occurs when the model makes a low perplexity (high confidence) prediction at a high entropy (high uncertainty) position, such as a proper name or a specific fact; on the contrary, at a low entropy (low uncertainty) position, such as a general language structure, the model's prediction lacks the ability to distinguish between members and non-members; therefore, by giving higher weights to high-entropy positions, the invention aims to amplify the most critical memory signals while suppressing noise interference from predictable language patterns, thereby significantly improving the signal-to-noise ratio and accuracy of the audit.

[0073] The information weight is obtained by normalizing the Shannon entropy values of all positions in the sequence, ensuring that the sum of all weights is 1, and its specific calculation formula is:

[0074]

[0075] Among them, w i H(p) represents the information weight at the i-th position. i ) is the Shannon entropy at the i-th position calculated in the first step, n is the total length of the text sequence to be audited, i.e. the total number of lexical units it contains, and the summation variable j iterates through all lexical positions from 1 to n.

[0076] E3: Aggregate the semantic perception density calculated at all positions in the entire text sequence with the corresponding information weights to calculate a single index that can comprehensively represent the model's overall memory of the sequence, namely "weighted semantic perception perplexity".

[0077] Aggregating local signals from all positions in the sequence into a single, robust audit score that represents the entire sequence, such as... Figure 7 As shown in the "Weighted Semantic Perception Confusion" module, this index is obtained by weighting and summing the negative logarithms of the semantic perception density at each position and taking its exponent:

[0078]

[0079] in, The final calculation result of the weighted semantic perception perplexity is a weighted summation process, which is determined by the position... The determined information weight w i With the real word t at that position i The negative logarithm of semantic perception density The products are multiplied and accumulated over the entire sequence length n. The accumulated result is then used as the input to the exponential function. A lower perplexity value indicates that the model is less "unexpected" about the sequence, i.e., more memorized.

[0080] E4: Compare the calculated weighted semantic perplexity with a pre-calibrated discrimination threshold, which is determined by running the same process on a large number of non-training data samples. If the weighted semantic perplexity of the text to be audited is lower than this discrimination threshold, the text sequence is determined to belong to the training data member of the target large language model.

[0081] To ensure the objectivity and adaptability of the judgment criteria, a discrimination threshold τ is determined through a statistical calibration process, which applies to a non-member calibration set. Each sample Calculate its weighted semantic perplexity to form an empirical distribution of non-member scores. Then, use a quantile function to select a lower quantile of this empirical distribution as the discrimination threshold τ.

[0082]

[0083] wherein q is a preset quantile hyper-parameter.

[0084] As Figure 7 In the "threshold comparison" module, the score of the text T to be audited is compared with the threshold τ during auditing. If , then the final judgment is that the text sequence to be audited is "training data".

[0085] In order to verify the effectiveness and advancement of the embodiments of the present scheme, the present scheme is compared with two mainstream baseline methods on an industry-recognized benchmark dataset, namely the Wikipedia benchmark dataset. The first method is based on traditional perplexity, and the second method is the current mainstream black-box auditing method which relies on a substitute model.

[0086] Table 1

[0087]

[0088] Table 1 is the comparison experimental result of the area under the receiver operating characteristic curve performance. Referring to Table 1, the table shows the area under the receiver operating characteristic curve (AUROC) performance of each method under different model sizes (Llama2-13B and Llama2-30B) and different text lengths. AUROC is an index for comprehensively measuring the performance of a classifier. The closer the value is to 1.0, the stronger the ability of the model to distinguish between positive and negative samples (i.e., training data and non-training data). From the data in the table, it can be seen that the "semantic-aware perplexity" method of the present application has achieved the highest AUROC score under all settings, which proves that the present application has significant superiority in overall discrimination ability.

[0089] Table 2

[0090]

[0091] ​Table 2 is the recall performance comparison experiment result under low false positive rate. Referring to Table 2, the table further compares the recall (True Positive Rate, TPR) performance under the condition of low false positive rate (False Positive Rate, FPR) which is more critical in practical application. In many audit scenarios (such as copyright confirmation, privacy leakage detection), the cost of misjudging non-members as members (false positives) is very high, therefore, an excellent audit method must strictly control the false positive rate while finding as many real members as possible. TPR measures the proportion of training data that the method can successfully identify under a given false positive rate. The experimental results clearly show that under the strict condition of controlling the false positive rate to 5%, the method of the present application can achieve significantly higher TPR, which fully demonstrates that the signal of the present application is more accurate and robust, and has higher reliability and practical value in practical application.

[0092] In S1, the target sequence refers to the input text to be audited, which is the continuous text (or a series of word units) submitted to the large language model through the text generation interface; the word unit position refers to the serial number or specific position of each specific word unit (i.e. the smallest text unit such as a word, symbol, character, etc.) in the target sequence; the successor candidate content refers to the set of word units generated in the next step when the model predicts based on the current context, each word unit is called a "candidate"; the occurrence frequency refers to the number of times each candidate word unit is generated in multiple sampling processes; the frequency change fluctuation amplitude refers to the degree of change or the size of the numerical fluctuation of the occurrence frequency of the same word unit in multiple samplings, which is used to measure the stability of the distribution; the content section refers to the continuous position fragment with similar distribution characteristics divided in the target sequence through continuous analysis of the word unit and its sampling distribution; the candidate content sampling distribution refers to the distribution of the occurrence probability or proportion of all candidate word units obtained by sampling multiple times for each word unit position; the fluctuation trend refers to the judgment of whether the sampling frequency of the section shows a stable, rising, falling or dramatic change state by comparing the distribution of different round sampling results.

[0093] In S2, the trend of change in context refers to the change trend of the frequency distribution when analyzing the current word and its previous and subsequent words (i.e. context), such as gradually changing from unstable to stable; the word with a smaller frequency fluctuation range refers to those positions whose frequency distribution fluctuates less, converges, and shows a stable trend after multiple rounds of sampling; the sampling position with higher frequency than other content refers to the position where the frequency of the word is significantly higher than other similar content during the sampling process of the candidate word, which usually reflects high confidence or bias of the model at the point; the convergence standard refers to the specific basis for judging whether the sampling distribution of a word position has reached a stable state (such as the distribution change amplitude being lower than the set standard); and the convergence of the change section refers to the description of whether the sampling distribution of all words in a specific section has reached a stable state as a whole or in part.

[0094] In S3, the real content refers to the word that should originally appear in the target sequence and actually exists in the input text (i.e. ground truth); the semantic path refers to whether there is a link of "semantic implication" or reasoning relationship between the candidate word and the real word, which can be one-hop, two-hop or multi-hop path; the semantic span refers to the distance between the candidate word and the real word in the semantic association network, which usually reflects the tightness or looseness of their semantic association; the intermediate content in the path refers to the word that is in the middle position and connects the two ends in the semantic path from the candidate word to the real word (similar to "bridge word" or "transfer word"); the order in the sampling content refers to the sequence characteristics such as the ordering position and frequency of the intermediate content in multiple sampling; and the path span refers to the minimum number of hops required for the semantic path from the candidate word to the real word, reflecting the distance between them.

[0095] In S4, the information change content position refers to the specific word position where the amount or distribution of information changes significantly through full sequence information analysis, which is usually related to the model "memory"; the change range of the adjacent content refers to the distribution, information amount, change and fluctuation interval of the adjacent content before and after a certain word; the contrast parameter of information distribution tension refers to a set of standard parameters used to evaluate and compare the strength of information amount change at each position, which is often used for difference comparison within the window interval; the window refers to a sub-sequence of words formed by expanding a certain word by a certain range in both directions (local analysis interval); the semantic support change refers to the change of semantic evidence strength between different words within the window interval, i.e. the fluctuation of the model's "semantic memory" signal in this region; the information fluctuation refers to the degree of change of the information amount (such as entropy) of each word in the window interval with the position; and the one-by-one calculation method refers to sequentially performing comparison, pairing, calculation and other operations on each pair of corresponding content in the window interval, rather than overall weighting or averaging.

[0096] In S5, the non-member reference content refers to a text sample or data known not to participate in training, used as a reference group to distinguish the differences between training and non-training data; the pairing mode refers to the specific operation mode of aligning, matching and comparing the features of the sample to be audited with the features of the non-member reference content one by one; the lower limit of the reference interval refers to the lower boundary of the index distribution obtained by statistical analysis in the non-member reference group, used to define abnormal or deviating phenomena; the determination boundary refers to the boundary standard for distinguishing the sample belonging category, usually determined by the above reference interval, quantile or statistical parameter; the boundary moves up refers to adjusting the determination boundary to a higher position, usually used when the sample feature is significantly higher than the reference lower limit, to appropriately increase the determination threshold; the distribution of sample confusion and semantic support refers to the feature distribution formed by the weighted integration of confusion signals and semantic support signals in the entire sequence range for the current text to be determined (reflecting the "memory" and "confidence" features of the model).

[0097] Please refer to Figure 2 The acquisition step of the sequence distribution fluctuation feature is specifically:

[0098] S111: Based on the text generation interface return content, analyze the context content corresponding to each word position in the target sequence, compare the appearance frequencies of each word in the sampling results after multiple rounds of sampling, calculate the number difference of each candidate word in each sampling round, judge the change of frequency distribution, and obtain the sampling frequency distribution sequence;

[0099] From the results returned by the text generation interface, the specific position of each word element in the submitted input text is extracted, and the range of the context in which each word element is located is marked, usually taking a certain number of word elements before and after as the local context content, such as taking five word elements before and after the target word element as the context window. Each word element and its context are numbered and saved in a sequence mapping table before execution, in order to match the position with the subsequent sampling results. Then, the text generation interface is called for the target sequence to perform multiple rounds of sampling, keeping the sampling settings consistent during the sampling process, such as using a fixed temperature and sampling range. In each round of sampling, the candidate content generated for each target word element position is recorded. The candidate content is compared with the original word element by position and recorded. The different word elements and their frequencies generated at each position in all rounds are counted and recorded. A data structure with word element position as the core is constructed, with each position corresponding to a group of candidate content and its frequency. After multiple rounds of sampling, the results of each round are arranged vertically according to the position number. The number of occurrences of each candidate word element in each round is recorded, and the difference in word element frequency between each position in each round is calculated to determine the fluctuation degree of the position in different rounds. For example, if the frequency of a position is low in the first few rounds and then rapidly increases, the fluctuation value of the position is considered large. Positions with long-term similar frequencies or small changes in frequency are considered to have weak fluctuations. A data sequence representing the frequency change trend of candidate content in sampling rounds is constructed for each word element position. This sequence is used for subsequent analysis to identify the stability of each position.

[0100] S112: Based on the sampling frequency distribution sequence, compare the frequency change data of each word element position, identify the continuous segment with a change amplitude in multiple samplings, judge the change trend and characteristics of the segment, and obtain a fluctuation segment label set;

[0101] Compare the change amounts of all positions in the entire target sequence with each other to determine which positions exhibit prominent change patterns. In the comparison process, the frequency change difference between adjacent two word elements is recorded. When a large continuous difference value appears at multiple positions, it is identified as a continuous segment with obvious changes. Then, trend analysis is performed on the change segment to observe whether the change is continuously rising, continuously falling, or fluctuating sharply between different rounds. The change is divided into types and labeled respectively according to the trend characteristics, for example, if the frequency gradually rises in a continuous segment, it is labeled as an upward trend, if the frequency gradually falls, it is labeled as a downward trend, and if the frequency changes randomly and disorderly between adjacent rounds, it is labeled as a sharp fluctuation. Next, each segment of continuous segment with obvious changes is bound with its trend type according to the position index, and the start position, end position, and change type of each segment are recorded in the result to form a label set specially used to describe the fluctuation region of the sequence. This set is used for subsequent focused distribution analysis of candidate content.

[0102] S113: Based on the fluctuation section label set, the candidate word frequency distribution corresponding to each section is calculated, the change trend and order of each position in the section are analyzed, the distribution change type in each section is judged, and the sequence distribution fluctuation feature is obtained;

[0103] The frequency analysis processing is sequentially performed on all word positions in each fluctuation section. In each section, the frequency data of all candidate contents of each position in multiple rounds of sampling is extracted, the data is compared horizontally to compare the distribution patterns of different words in adjacent positions, the ranking of candidate contents in each position in each round of sampling is counted, and the ranking stability between multiple positions is calculated. If the candidate rankings of adjacent positions are basically consistent, the section ranking is stable, if the ranking difference is large, the section fluctuation is significant, and the candidate content is concentrated in a few high-frequency words. The position is marked, such as a position with only one or two candidate words with high frequency, and the frequency of the remaining candidate words is much lower than the main candidate. The position is considered as a concentrated distribution. If the frequency of all candidate words is similar and changes significantly in different rounds, the position is considered as a fluctuation distribution. Through the statistics of the ranking and frequency change of the entire fluctuation section, the specific distribution type of the section is judged, and the section range, ranking stability, distribution fluctuation degree and other information are recorded. A complete sequence distribution fluctuation feature set is formed, which is used to further judge whether the model shows memory or sensitivity characteristics to certain content at the semantic level.

[0104] Please refer to Figure 3 The convergence index set is obtained by the following steps:

[0105] S211: Based on the sequence distribution fluctuation feature, the frequency distribution change of each word in the context window is analyzed, the distribution sequence of candidate content of each word in multiple rounds of sampling is judged, the positions with gradually slowing changes and gradually reduced differences are identified by comparing the distribution convergence of each sampling result, and a convergence position index set is obtained.

[0106] The frequency distribution of each word in its corresponding context window is analyzed one by one. First, the position of the target word in the sampling sequence is located, then a fixed-length context window is formed by extending five positions forward and backward, and all candidate words and their corresponding frequencies that appear in the word in multiple rounds of sampling are extracted in the window. The candidate words generated at this position in each round of sampling are taken as a set, and the distribution sequence corresponding to the word is established. Each round of sampling result corresponds to a candidate set. By comparing the repetition rate of high-frequency words in each round of candidate set, it is recorded whether the high-frequency words in the previous and next rounds are stable. If the number of high-frequency words decreases and the number of occurrences is concentrated in continuous rounds, it is determined that the candidate distribution of the position tends to converge. For example, four high-frequency candidate words appear at a certain word position in the first round, with similar frequencies. In the second round, only two candidate words appear at this position, and the frequency is obviously dominant. In the third round, one of the candidate words has a frequency of more than 70% of the total number of samples. The position converges obviously. Then, the amplitude of the distribution change of each position in all sampling rounds is calculated. When the number of word changes between consecutive rounds is less than 3, and the repetition rate of the highest frequency word between the previous and next rounds is more than 80%, it is determined that the distribution of the position has tended to be stable. Each position is traversed in turn and the position index that meets the stable condition is recorded to form a convergence position index set, which is used to represent which words show stable characteristics in the frequency change in the sampling process.

[0107] S212: Based on the convergence position index set, compare the candidate word frequency distribution corresponding to each position with the aggregation characteristics of adjacent positions, judge the distribution difference of the internal aggregation density of the section, identify the section in the sampling frequency aggregation feature set, and obtain the aggregation feature section group;

[0108] The candidate word frequency data corresponding to each convergence position is extracted one by one, and the data of the adjacent word position before and after it is aggregated. The current position and the two adjacent positions above and below form a local comparison region. The candidate frequencies of all word elements in the region are combined and counted. The cumulative frequency of each word element in the small region is analyzed and ranked in descending order. The aggregation density is judged according to the word element frequency proportion. If one or two word elements have cumulative frequency proportions exceeding 70% of the total frequency, the position is determined as a frequency aggregation position. Otherwise, if no word element cumulative frequency proportion exceeds 40%, the position is determined as a frequency dispersion position. Next, the number distribution of frequency aggregation in all convergence positions is counted. If three or more consecutive convergence positions are all aggregation positions, the position segment is marked as a high-density aggregation segment. The start position, end position and main distribution form of the aggregation word element in the segment are recorded. For example, in actual sampling, if the 8th to 10th positions are all dominated by the word element "way" or "method", and the frequencies of other candidate word elements are far lower than half of the dominant word element, then the three positions are recorded as a group of aggregation feature segments. By classifying the local aggregation of all convergence positions, multiple segment structures are formed, and an aggregation feature segment group is constructed to depict the sequence region with high concentration of candidate word elements in the semantic generation process.

[0109] S213: Based on the aggregation feature segment group, the candidate word frequency distribution structure of the corresponding segment is calculated, the convergence performance and sequence attribution trend of each position in the segment are analyzed, the distribution classification of the aggregation segment is judged, and the convergence index group is obtained.

[0110] The frequency of each candidate word in each section is counted, the frequency distribution of all positions in the section is traversed and compared horizontally, and it is confirmed whether each word in the section shows a stable high-frequency state. In the analysis process, the frequency value of each word in each position is aligned, and the frequency proportion of the same word in different positions is compared. If the word is the highest frequency word in most positions and the frequency proportion is more than 60%, it is determined that the word has a convergence advantage in the section, and the word is recorded as the main representative word of the section. Continue to determine whether the entire section has a single attribution trend. If the main words of multiple positions in the section are the same content and have obvious frequency advantage, the section is classified as a single attribution section. Otherwise, if the main words of different positions in the section change frequently and the frequency difference is small, it is recorded as a mixed attribution section. For example, a certain aggregation section consists of positions 12 to 15, where the highest frequency words of positions 12, 13 and 15 are all "strategy", and the position 14 is "method". If "strategy" appears 3 times out of 4 times in the section, and the average frequency proportion is more than 65%, the section can be classified as a single attribution section dominated by the "strategy" word. The classification results of all aggregation feature sections are summarized, and the section position, dominant word and attribution state are structured to output the convergence index group, which is used as the basis for candidate path construction in the subsequent semantic evidence extraction stage.

[0111] Please refer to Figure 4 The acquisition steps of the semantic evidence weight set are as follows:

[0112] S311: Based on the convergence index group, the corresponding relationship between each candidate word and the true word in the semantic space is analyzed, the path structure of the semantic link between the candidate content and the true content is compared, the word order of each jump link in the path is judged, and the semantic link structure set is obtained.

[0113] Firstly, the actual real word element appearing at each convergence position is determined, and all candidate word elements generated in all sampling rounds corresponding to the position are extracted. Each convergence position is traversed in turn, and the real word element is combined with all candidate word elements at the position in pairs. Each pair of combination is regarded as a set of semantic corresponding items to be analyzed. Then, the vector expression of each word element is obtained from the semantic embedding space. The coordinate points of the candidate word element and the real word element in the embedding space are connected. The jump relationship of the connection path is identified. The starting point of each path is defined as the candidate word element, the ending point is the real word element, and the intermediate word element in the path is the word element that frequently appears in multiple sampling rounds and has an indirect semantic association with the target content. It is judged whether the intermediate word element appears at the adjacent position of the convergence index group. If the high-frequency appearance and the continuous semantic distance conditions are met at the same time, the jump link is included. The order mark of the intermediate word element of each jump link is executed. The position number of the first appearance of the intermediate word element in the sampling sequence is recorded. The path from each candidate to the real content and the jump point in the path form a sequential link structure. The jump order table of the path is constructed. For example, the real word element is "strategy", and the candidate word elements are "method", "scheme" and "mechanism". If all of them frequently appear in the sampling rounds, "method" is located in the first two positions of "strategy", and "mechanism" is located in the last two positions, then the path jump order is "method"→"scheme"→"strategy". Further, the above information is sequentially stored in the structure data table to form a semantic link structure set with the candidate to real word element path as the core.

[0114] S312: Based on the semantic link structure set, the semantic span of each path is compared, the semantic distance from the starting point to the ending point of each path and the distribution change of the jump link content are calculated, the parameter relationship between the span and the jump link characteristics is analyzed, and the semantic span mapping data is obtained;

[0115] The complete path of each candidate word element in each link to the real word element is extracted one by one, the jump span of each jump link in the path in the embedding space is calculated, and the total amount of semantic interval between all jump sections in the path is recorded. The starting word element, all intermediate word elements and the termination word element of each path are numbered in the order of arrangement in the semantic space, and the jump content is recorded as a node sequence. The start-to-end distance between different paths is compared longitudinally, and whether the span value is concentrated in a certain range is counted. For example, if the path span is between 2 and 3 intermediate word elements, the proportion of paths reaches 70%, then the span type is marked as medium span path. The word element content of each jump link is compared to analyze the relationship between semantic similarity and position, and to determine whether the candidate content with closer word meaning to the real word element is closer to the path end point. At the same time, it is observed whether the word element in the jump link has the characteristic of frequent cross-path appearance. If a word element appears in more than three paths and the appearance position is concentrated in the second last position, it is marked as an end transition word element. By integrating the span value, jump word element distribution and context position number in each path, the parameter corresponding relationship between semantic span and node content in path structure is summarized, and path span mapping data is formed, which is used as a basis for subsequent sorting and weight fusion stage.

[0116] S313: Based on the semantic span mapping data, the ordering position of the intermediate word element of each path in the sampling sequence is determined, the distribution rule of the ordering data and the path span characteristics is analyzed, the fusion mode of the ordering parameter and the span parameter is optimized, and the semantic evidence weight set is obtained.

[0117] The prepositions in each path are read in sequence, and the first occurrence position and average occurrence order position of the word element in all sample sequences are searched, the average value of the relative position in all sample rounds is counted for each preposition, and the average position is sorted in ascending order, whether the appearance position in the sample sequence is consistent with the sorting in the path is recorded, if the consistency is more than 80%, it is determined that the path sorting stability is high, then the sorting data and the path span data are combined, the ranking of the sorting of each preposition and the semantic jump distance in the path are calculated, whether the word element with high ranking and far from the end point has strong intermediate signal is observed, if the sorting is in the front but the semantic span is lower than the average value of the path, the word element is marked as a high-ranking low-span node, the frequency of the node appearing in the path is counted, if the proportion is more than 60% of the number of path nodes, the path is classified as a short-span preposition path, otherwise, if the sorting is in the back and the semantic span is high, it is classified as a long-span postposition path, finally, the sorting deviation value and the semantic span value are normalized to generate the sorting weight and the span weight corresponding to the word element, and the two types of weights are fused by weighted average, maximum value selection or threshold screening, if the sorting deviation exceeds the set value 3 during the fusion process, the corresponding word element weight is directly excluded, the fusion weight of the remaining word element is standardized so that the sum is 1, the weight value, path number and occurrence position of each preposition are recorded to form a semantic evidence weight set.

[0118] Please refer to Figure 5 The tension comparison signal group acquisition step is specifically:

[0119] S411: Based on the semantic evidence weight set, analyze the information distribution characteristics of each word element in the sequence, judge the change trend of the semantic support parameter, compare the fluctuation performance of the information intensity at each word element position, identify the content points showing information mutation or consistency reduction in the distribution, and obtain a variation feature index;

[0120] The specific position of each word unit in the target sequence is extracted, and the corresponding semantic evidence weight is combined. The sequence is scanned in position order. The weight value of each word unit is read during the scanning process, and the weight change difference between adjacent positions is recorded. The semantic weight values of all word units in each window are summarized to form a local information distribution map. The information intensity values corresponding to all word units in the map are plotted as a one-dimensional vector. Whether there are value mutations or concentrated fluctuations in the continuous region of the vector is observed. Each continuous change point is judged. When the weight value of any word unit position changes more than 150% of the average value between the two adjacent word unit positions above and below it, the position is determined to be an information mutation point. If the fluctuation range of the semantic weight of the continuous three or more positions is less than 20% of the average fluctuation range of the entire sequence, the segment is marked as a consistency weakening section. Further, all word unit positions that meet the above mutation or consistency decrease conditions are collected and formed into a set in position number, such as in a certain sampling result, the weight value of the 16th position is 0.91, and the weights of the 15th and 17th positions are 0.43 and 0.45, respectively. The 16th position meets the mutation condition and is recorded. At the same time, the weight values of the 21st to 23rd positions are 0.31, 0.30, and 0.32, respectively, and the overall fluctuation is less than the average fluctuation standard of the sequence, so the three-position section is recorded as a consistency weak point segment. The position information of all mutation points and consistency reduction points is sorted and summarized to form a variation feature index, which is used as input reference for subsequent semantic tension analysis steps.

[0121] S412: Based on the variation feature index, the semantic distribution range of the target position and adjacent word units is judged, the continuity of the semantic structure before and after each position is compared, the offset performance of the context distribution is analyzed, the information fluctuation and distribution difference segment is identified, the position is divided into intervals, and the segment fluctuation window is obtained.

[0122] The semantic distance between the variant position and the adjacent position is compared by calling the word embedding information in the context window range of each position. If the distance is less than the set threshold, it is considered that the two positions are semantically continuous. If the distance exceeds the set threshold, it is determined that the semantic is broken. The semantic distance of the continuous position pairs is compared by pair, and it is judged whether there is a whole shift in the semantic structure in the region. If multiple variant positions appear in the context and form a fragment with significantly expanded or reduced semantic distribution span, it is recorded as a context offset interval. For example, in the actual sequence, the 10th to 14th positions were originally a semantically continuous fragment, but the 12th position was recorded due to information mutation, and the semantic vector cosine difference between the 12th position and the 11th and 13th positions exceeded 0.6. It is considered as a broken point and the original fragment is split. Then the context area connected by the broken point is re-divided into segments. In each divided segment, the semantic relationship strength of all words in the interval is aggregated and analyzed, and the semantic jump frequency and average interval difference between the front and rear positions are counted. If the jump frequency is dense or the interval difference is greater than twice the average level of the whole sequence, the segment is marked as an information fluctuation fragment. All fragments are numbered and divided according to semantic continuity to form multiple semantic difference windows, such as a window from the 5th to the 9th position and another window from the 11th to the 14th position. The number of words contained in each window, the average semantic change value and the boundary position are recorded. The window number and position information are output to form a segment fluctuation window data structure for subsequent tension signal analysis.

[0123] S413: Based on the segment fluctuation window, call semantic support data and information fluctuation parameters to analyze the change direction and structural characteristics of each group of windows. Judge the pairing difference between the signals in the same group to get the tension comparison signal group.

[0124] The semantic support data and information fluctuation parameters of all word units in each window range are extracted in turn, and two signal sequences in the window are established respectively. The first signal sequence is the semantic support value sequence, which is taken from the values of the corresponding word units in the semantic evidence weight set. The second signal sequence is the information fluctuation intensity sequence, which is taken from the semantic jump difference values of the word units and the upper and lower word units in the semantic space. The difference direction of each two adjacent word units in the two signal sequences is calculated in the window, and its change type is marked as rising or falling. When the change directions of the same position pairs in the two signal sequences are inconsistent, the point is recorded as a signal confrontation point. If signal confrontation occurs for three or more consecutive word units, the window is determined as a high tension window. Conversely, if the synchronous rising or falling proportion of the two signals exceeds 80% of the total number of positions in the window, the window is determined as a low tension block. Further, the difference value size of all signal pairs in the same window is counted. If the maximum difference value is higher than twice the average difference value in the window, the difference point is recorded as a key tension contrast point, and the sequence offset degree of the position in the window is calculated. When the offset degree is close to the window boundary position, it is recorded as a boundary tension point, forming a structured tension contrast signal group, which is used as the input parameter set of semantic attribution analysis.

[0125] Please refer to Figure 6 The acquisition step of the attribution adaptation determination result is specifically:

[0126] S511: Based on the tension contrast signal group, the pairing structure of the non-member reference content in the current sample and the training sample is analyzed, the distribution performance of each pairing signal in the difference sample is compared, the influence of the pairing mode on the matching result between signals is judged, and the signal pairing configuration is obtained.

[0127] The tension signal features formed by the current sample to be analyzed at all word position are structurally paired with the tension signals of the known non-member reference content in the training sample, the difference value, offset position, signal pairing direction recorded in each tension signal pair in the current sample are extracted and aligned with the corresponding items in the reference sample, whether there is a matching item is recorded for each pairing result, if there is, the size of the difference value and the offset direction are compared, if there is not, it is treated as missing, then each group of paired signals is classified according to the structure label, the proportion of the current sample that is completely matched with the non-member sample in all signal pair structures is counted, if the matching proportion exceeds 80%, the sample has high structural similarity with the reference set, if the matching proportion is less than 40%, the structural difference between the two is large, then the key position points in the signal pairing are extracted as the main axis of the structure, the pairing density on both sides of the main axis is counted, the distribution tendency of the paired signals in the sample structure is judged, if the main paired signals are concentrated in the first 40% or last 40% position, it is marked as a structure uneven pairing mode, if the paired signals are uniformly distributed in the middle segment and the distribution standard deviation is less than 5%, it is marked as a stable pairing structure, on this basis, difference value analysis is performed on all signal pairs, items with difference value more than twice the median difference value of the reference sample are marked as structure confrontation points, then the proportion of such confrontation points in all signals is counted, if the proportion exceeds 20%, the pairing mode is recorded as a tension distribution offset type structure, the matching density, difference intensity, main axis concentration, pairing position distribution and confrontation point proportion of the pairing structure are integrated as the components of the signal pairing configuration.

[0128] S512: Based on the signal pairing configuration, the matching degree between the current sample signal and the non-member reference content signal is judged, the attribution of each type of signal matching at the interval boundary is compared, the difference between the distribution characteristics is analyzed, and the sample classification parameter group is obtained.

[0129] The matching degree is calculated item by item after comparing all the tension pairing signals in the current sample with the corresponding signals in the non-member reference content, recording the type, position and intensity of each set of pairing signals in the structural configuration. The matching degree is calculated by counting whether the pairing direction of each signal is consistent, whether the difference value is within the tolerance range, and whether the position is coincident. When all three factors are met, it is recorded as complete matching, otherwise it is partial or non-matching. After summarizing all the pairing signals, the total matching ratio is calculated. If the matching ratio falls between 60% and 80%, it is considered to be intermediate matching. If it exceeds 80%, it is considered to be high matching. If it is less than 40%, it is considered to be low matching. At the same time, all matching results are sorted by section. The aggregation trend of pairing signals is judged at the boundary of different sections. If most of the pairing signals are concentrated within five word units before and after the boundary, the signal matching structure is classified as boundary bias type. If the pairing signals span multiple sections but are evenly distributed in each section, it is classified as multi-section balanced type. The position distribution and difference of different signal types in the above matching structure are further analyzed. For example, the concentration degree of strong tension signals and weak tension signals in high matching and low matching areas is calculated respectively. If strong signals are mainly concentrated in high matching areas, while weak signals are widely distributed, it indicates that the current sample has a pairing dominant tendency. The matching statistics, distribution range and boundary attribution of all types of signals in the pairing structure are summarized to form a sample classification parameter group for classification and judgment.

[0130] S513: Based on the sample classification parameter group, calculate the sample confusion parameter and semantic support parameter distribution, analyze the distribution characteristics of signal types in the full sequence range, judge the attribution type and probability interval, and obtain the attribution adaptation judgment result;

[0131] The semantic support parameter and the perplexity parameter of the current sample in the whole word sequence are extracted, the whole sequence of the sample is traversed, the corresponding semantic evidence weight is recorded as the semantic support value for each word position, the perplexity value is calculated through the frequency distribution fluctuation of the target word in the multi-round sampling process, two corresponding sequence signal vectors are constructed, the relative difference of the two types of signals in each interval is counted, and it is judged whether the perplexity value is mainly concentrated in the low semantic support area in the overall structure of the sample. If the concentration proportion exceeds 70%, it is considered as a weak support and high perplexity structure, otherwise if the perplexity value is concentrated in the position where the semantic support value is more than 0.8, it is classified as a high support and high perplexity type. According to the signal matching degree and structure characteristics recorded in the sample classification parameter group, the belonging state of the current sample is judged, the range of the perplexity signal and the range of the semantic support are compared, if the difference between the maximum values of the two is less than 0.1, it is classified as a boundary overlapping sample, if the median of the semantic support is higher than the median of the perplexity parameter and the difference between the two is greater than 0.2, it is classified as a support dominant sample. Further map the attribution result to the set probability interval, for example, the high matching support dominant sample is allocated to the attribution probability interval of 80%-100%, and the low matching and perplexity dominant sample is allocated to the interval of 20%-40%, the attribution label, interval range and signal proportion information of each sample are recorded to form the attribution adaptation judgment result.

[0132] A semantic-aware black-box large language model training data auditing system, the system comprising:

[0133] The fluctuation feature analysis module returns the content based on the text generation interface, obtains the candidate content through multi-round sampling, compares the frequency of each word content, identifies the section with frequency fluctuation in continuous sampling, determines the fluctuation trend, and obtains the sequence distribution fluctuation feature;

[0134] The convergence determination module selects the word with frequency fluctuation trend to be stable as the convergence point based on the sequence distribution fluctuation feature, adjusts the convergence standard for high-frequency sampling position, re-determines the convergence of the change section, and obtains the convergence index group;

[0135] The semantic weight calculation module calculates the semantic path between the real content and the candidate content based on the convergence index group, analyzes the semantic span and the order of the intermediate content in the path, integrates the path span and order information, and obtains the semantic evidence weight set;

[0136] The tension pairing module selects the content position with information change based on the semantic evidence weight set, judges the content change range before and after the content position, and pairs the semantic support change and information fluctuation in the window one by one to obtain the tension comparison signal group;

[0137] The attribution analysis module adjusts the matching mode with the non-member reference content based on the tension contrast signal group, analyzes the difference between the current sample and the reference content, optimizes the judgment boundary according to the matching degree, and judges the distribution of confusion and semantic support, to obtain an attribution adaptive judgment result.

[0138] The unmentioned parts of the present application apply to the prior art.

Claims

1. A semantic-aware black-box large language model training data auditing method, characterized in that, The method comprises the following steps: S1: based on the text generation interface returning content, obtaining candidate content through multiple rounds of sampling, comparing the occurrence frequency of each word element content, identifying the section with frequency fluctuation in continuous sampling, determining the fluctuation trend, and obtaining the sequence distribution fluctuation feature; S2: based on the sequence distribution fluctuation feature, filtering the word element with frequency fluctuation trend to be stable as the convergence point, adjusting the convergence standard for high-frequency sampling position, re-determining the convergence condition of the change section, and obtaining the convergence index group; S3: based on the convergence index group, calculating the semantic path between the real content and the candidate content, analyzing the semantic span and the order of the intermediate content in the path, integrating the path span and order information, and obtaining the semantic evidence weight set; S4: based on the semantic evidence weight set, filtering the content position with information change, judging the content change range before and after the content position point, pairing the semantic support change and information fluctuation in the window one by one, and obtaining the tension comparison signal group; S5: based on the tension comparison signal group, adjusting the pairing mode with the non-member reference content, analyzing the difference between the current sample and the reference content, optimizing the determination boundary according to the matching degree, and judging the distribution of confusion and semantic support, and obtaining the attribution adaptation determination result.

2. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The sequence distribution fluctuation feature comprises a distribution stability label, a fluctuation critical point and a trend recognition parameter, the convergence index group comprises a position identifier, a stability level and a standard adjustment factor, the semantic evidence weight set comprises a weight distribution factor, a coverage coefficient and a path mapping index, the tension comparison signal group comprises a signal difference amount, a section change coefficient and a pairing result label, and the attribution adaptation determination result comprises an attribution classification, a level label and a probability interval.

3. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The acquisition step of the sequence distribution fluctuation feature is: S111: based on the text generation interface returning content, analyzing the context content corresponding to each word element position in the target sequence, comparing the occurrence frequency of each word element in the sampling result after multiple rounds of sampling, calculating the number difference of each candidate word element in each sampling round, judging the change of frequency distribution, and obtaining the sampling frequency distribution sequence; S112: based on the sampling frequency distribution sequence, comparing the frequency change data of each word element position, identifying the continuous section with change amplitude in multiple samplings, judging the change trend and feature of the section, and obtaining the fluctuation section label set; S113: based on the fluctuation section label set, calculating the frequency distribution of the candidate word element corresponding to each section, analyzing the change trend and order of each position in the section, judging the distribution change type in each section, and obtaining the sequence distribution fluctuation feature.

4. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The acquisition step of the convergence index group is: S211: based on the sequence distribution fluctuation feature, analyzing the frequency distribution change of each word element in the context window, judging the distribution sequence of the candidate content of each word element in multiple samplings, identifying the position with gradually slowing change and gradually reducing difference by comparing the distribution convergence condition between sampling results, and obtaining the convergence position index set; S212: Based on the convergence position index set, compare the candidate word frequency distribution corresponding to each position with the aggregation characteristics of adjacent positions, judge the distribution difference of the internal aggregation density of the section, identify the section in the sampling frequency aggregation feature set, and obtain the aggregation feature section group; S213: Based on the aggregation feature section group, calculate the candidate word frequency distribution structure of the corresponding section, analyze the convergence performance and sequence attribution trend of each position in the section, judge the distribution classification of the aggregation section, and obtain the convergence index group.

5. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The acquisition step of the semantic evidence weight set is: S311: Based on the convergence index group, analyze the corresponding relationship between each candidate word and the real word in the semantic space, compare the path structure of the semantic link between the candidate content and the real content, judge the word order of each jump link in the path, and obtain the semantic link structure set; S312: Based on the semantic link structure set, compare the semantic span of each path, calculate the semantic distance from the starting point to the ending point of each path and the distribution change of the jump link content, analyze the parameter relationship between the span and the jump link characteristics, and obtain the semantic span mapping data; S313: Based on the semantic span mapping data, judge the ordering position of the intermediate word in each path in the sampling sequence, analyze the distribution law of the ordering data and the path span characteristics, optimize the fusion mode of the ordering parameters and the span parameters, and obtain the semantic evidence weight set.

6. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The acquisition step of the tension comparison signal group is: S411: Based on the semantic evidence weight set, analyze the information distribution characteristics of each word in the sequence, judge the semantic support parameter change trend, compare the fluctuation performance of information intensity at each word position, identify the content points that present information mutation or consistency reduction in the distribution, and obtain the variation feature index; S412: Based on the variation feature index, judge the semantic distribution range of the target position and the adjacent word, compare the continuity of the semantic structure before and after each position, analyze the offset performance of the context distribution, identify the segment of information fluctuation and distribution difference, divide the position by interval, and obtain the section fluctuation window; S413: Based on the section fluctuation window, call the semantic support data and information fluctuation parameters, analyze the change direction and structure characteristics of the two types of signals in each group of windows, judge the pairing difference between the signals in the same group, and obtain the tension comparison signal group.

7. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The acquisition step of the attribution adaptation determination result is: S511: Based on the tension comparison signal group, analyze the pairing structure of the non-member reference content in the current sample and the training sample, compare the distribution performance of each paired signal in the difference sample, judge the influence of the pairing method on the matching result between the signals, and obtain the signal pairing configuration; S512: Based on the signal pairing configuration, judge the matching degree between the current sample signal and the non-member reference content signal, compare the attribution of each type of signal matching in the interval boundary, analyze the difference between the distribution characteristics, and obtain the sample classification parameter group; S513: Based on the sample classification parameter group, calculate the sample confusion parameter and the semantic support parameter distribution, analyze the distribution characteristics of the signal type in the full sequence range, judge the attribution type and the probability interval, and obtain the attribution adaptation determination result.

8. The semantically-aware, black-box large language model training data auditing method of claim 1, wherein, The fluctuation trend refers to judging whether the sampling frequency of the section presents a stable, rising, falling or dramatic change state by comparing the distribution of each round of sampling results, and the convergence of the change section refers to the description of whether the sampling distribution of all word elements in the target section as a whole or locally reaches a stable state.

9. A semantically-aware black-box large language model training data auditing system, characterized in that, The system is used to implement the semantic-aware black-box large language model training data auditing method of any one of claims 1-8, and the system comprises: The fluctuation feature analysis module returns content based on the text generation interface, obtains candidate content through multiple rounds of sampling, compares the appearance frequency of each word element content, identifies the section with frequency fluctuation in continuous sampling, determines the fluctuation trend, and obtains the sequence distribution fluctuation feature; The convergence determination module filters the word elements with fluctuation trend tending to be stable as convergence points based on the sequence distribution fluctuation feature, adjusts the convergence standard for high-frequency sampling positions, re-determines the convergence of the change section, and obtains a convergence index group; The semantic weight calculation module calculates the semantic path between the real content and the candidate content based on the convergence index group, analyzes the semantic span and the order of the intermediate content in the path, integrates the path span and order information, and obtains a semantic evidence weight set; The tension pairing module filters the content positions with information changes based on the semantic evidence weight set, judges the content change range before and after the content position point, pairs the semantic support change and information fluctuation in the window one by one, and obtains a tension comparison signal group; The attribution analysis module adjusts the pairing mode with the non-member reference content based on the tension comparison signal group, analyzes the difference between the current sample and the reference content, optimizes the determination boundary according to the matching degree, and judges the distribution of confusion and semantic support, and obtains an attribution adaptation determination result.

Images