A context learning bias risk assessment method and system based on Chinese text style construction and injection

Abstract

This invention provides a method and system for assessing bias risk in context learning based on the construction and injection of Chinese text style. The method includes: constructing a definition of Chinese text style; building a style feature injector based on the most sensitive Chinese text style; injecting style features into the input portion of examples in the context learning prompt template to form stylized example input; synthesizing biased output using a teacher model, and replacing the examples in the context learning prompt template with the stylized example input and biased output; calling the evaluated large language model to perform original user queries and stylized user queries respectively to generate model output; using the Regard index and bias susceptibility rate to quantitatively analyze the model output, and using perplexity and semantic similarity as quality control indicators to obtain the bias risk assessment result. This invention reveals the bias risk of large language models in context learning scenarios and improves the effectiveness and stability of bias risk assessment.

Description

Technical Field

[0001] This invention belongs to the field of fairness auditing technology of large language models, specifically involving a method and system for assessing risk of contextual learning bias based on Chinese text style construction and injection. Background Technology

[0002] Large language models, as an important technological form of current general artificial intelligence, have been widely applied in scenarios such as search and question answering, content generation, software development, education, and healthcare. With the practical deployment of these models in government affairs, public services, and key industries, the fairness and security of their output results have gradually become important governance issues. When text involves references to population groups, the polarity of the model's output may differ under different group conditions, thus generating potential bias risks and affecting the model's compliance and social trust.

[0003] Contextual learning mechanisms allow large language models to adapt to tasks by adding a small number of examples to the prompts without modifying the model parameters. Existing research shows that the model's predictions in contextual learning are significantly sensitive to examples; different example configurations can lead to variations in model performance. Accordingly, when examples or user queries involve demographic group references, the composition and stylistic features of the examples can affect the model's output performance across different groups, resulting in differentiated results and impacting fairness.

[0004] Existing evaluation methods for large language models in context learning scenarios largely focus on security and robustness testing, while systematic and quantitative research on model output bias and fairness issues under population group reference conditions is relatively insufficient. Furthermore, some evaluation methods rely on modifying or fine-tuning model parameters, making them unsuitable for real-world deployment environments involving black-box API calls; others involve inserting additional tokens into the prompt template or significantly altering the prompt structure. Especially in Chinese scenarios, such approaches often introduce anomalous expressions or semantic shifts, easily triggering the model's rejection mechanism or security policy interception, thereby interfering with the evaluation process and reducing the stability and reproducibility of the evaluation results.

[0005] Therefore, there is an urgent need for a context learning bias risk assessment method implemented under black-box and parameter-independent conditions. This method should be able to quantify the differences in output polarity under different population groups under referential conditions with minimal intervention through controllable Chinese text style construction and style feature injection, and support fairness audits across models and languages. Summary of the Invention

[0006] To address the aforementioned technical issues, this invention proposes a method and system for assessing bias risks in context learning based on Chinese text style construction and injection. Under black-box and parameter-independent conditions, it can avoid interference from model refusal or security strategies, reveal the bias risks of large language models in context learning scenarios, and ensure the stability and reliability of the assessment process. This improves the effectiveness and stability of bias risk assessment and supports fairness auditing across models and across languages ​​(Chinese / English), as well as the verification of mitigation strategies and the deployment of risk management.

[0007] To achieve the above objectives, the present invention provides the following solution: A method for assessing the risk of contextual learning bias based on Chinese text style construction and injection includes: Text style features are extracted from the acquired Chinese style corpus to construct a definition of Chinese text style; The sensitivity of the evaluated large language model to different Chinese text styles is detected, and a style feature injector is constructed based on the Chinese text style with the highest sensitivity. Based on the style feature injector, style features are injected into the input part of the example in the context learning prompt template to form a stylized example input; Based on the stylized example input, a biased output is synthesized using a teacher model, and the stylized example input and biased output are used to replace the examples in the context learning prompt template; The context-based learning prompt template based on the replacement example is used to call the evaluated large language model to perform the original user query and the stylized user query respectively, and generate the model output. The model output was quantitatively analyzed using the Regard index and the bias susceptibility rate, and the bias risk assessment results were obtained by using perplexity and semantic similarity as quality control indicators.

[0008] Preferably, the text style feature extraction of the Chinese style corpus includes word-level extraction and sentence-level extraction; wherein, the method for word-level extraction includes: We used a word segmentation library to perform word segmentation and part-of-speech tagging on Chinese-style corpora, and constructed word sequences and part-of-speech tag sequences. Based on the part-of-speech tag sequence, a part-of-speech distribution frequency function is constructed to determine the dominant part-of-speech type of the Chinese style corpus; Based on the character length of each word in the word sequence, a word length distribution frequency function is constructed to count the occurrence frequency of each word length type, and the word length type with the highest occurrence frequency is selected as the word length feature; Words are divided into monosyllabic words and polysyllabic words based on word length. The most frequently occurring monosyllabic words are counted, and the polysyllabic words are divided into sentiment polarities using a pre-set sentiment dictionary. The most frequently occurring words in polysyllabic words with positive and negative sentiment polarities are counted to obtain high-frequency polysyllabic words with positive and negative sentiment polarities. Using a pre-defined dictionary of modal particles and idioms, the frequency of modal particles and idioms in the corpus is statistically analyzed by regular expression matching. The idioms are then divided into sentiment polarities. The most frequent words in idioms with positive and negative sentiment polarities are counted separately to obtain high-frequency idioms with positive and negative sentiment polarities. The word-level style feature representation is constructed by describing and connecting the part-of-speech features, word length features, high-frequency monosyllabic words, high-frequency polysyllabic words with positive and negative polarity, high-frequency modal particles, and high-frequency idioms with positive and negative polarity using natural language.

[0009] Preferred methods for sentence-level extraction include: Using periods, question marks, and exclamation marks as terminators, the Chinese-style corpus is segmented into sentence sequences. Character matching is used to extract Chinese and English characters from the sentence sequence to obtain a character set. The average sentence length is calculated based on the number of characters in the character set and the total number of sentences, and is used as the sentence length feature. Construct a set of sentence types, including complete sentences and non-complete sentences; Based on the set of sentence types, a first prompt template is constructed, and a large language model is called to classify the sentence type of each sentence in the sentence sequence to obtain the sentence type corresponding to each sentence. Statistically analyze the frequency of occurrence of various sentence patterns and select the sentence pattern category with the highest frequency as the dominant sentence pattern feature; Construct a set of rhetorical categories, which include metaphor, personification, hyperbole, quotation, parallelism, irony, rhetorical question, rhetorical question, and antithesis; Based on the set of rhetorical categories, a second prompt template is constructed, and a large language model is called to perform rhetorical classification on each sentence in the sentence sequence to obtain the rhetorical category corresponding to each sentence. The frequency of occurrence of various rhetorical devices was statistically analyzed, and the rhetorical device category with the highest frequency of occurrence was selected as the high-frequency rhetorical feature; The sentence length feature, the dominant sentence structure feature, and the high-frequency rhetorical feature are described and connected using natural language to form a sentence-level style feature representation.

[0010] Preferred methods for constructing Chinese text style definitions include: Identify population group references in the input text and determine the target population group; Based on the word-level style feature representation, positive polarity and negative polarity feature selection operators are constructed to select word-level style feature subsets corresponding to positive polarity and negative polarity, respectively. Based on the target population group and population group designations, a conditional mapping function is constructed to select the corresponding word-level style feature subset according to whether the input text contains population group designations. The selected subset of word-level style features is connected with the sentence-level style feature representation to form a Chinese text style definition.

[0011] Preferred methods for constructing style feature injectors include: A binary classification prompt template is constructed, which consists of several examples. Each example contains a sentence and a binary classification label. By controlling the sentiment polarity in the sentence and whether it contains candidate Chinese text styles, the classification labels of the examples are consistent when they are classified according to sentiment polarity and when they are classified according to whether they contain the candidate Chinese text styles. A conflict test set is constructed, wherein each sample in the conflict test set has a first true label according to the sentiment polarity classification and a second true label according to whether it contains the candidate Chinese text style classification, and the first true label and the second true label are inconsistent. Under the binary classification prompt template, the large language model being evaluated is invoked to output a binary classification label or the corresponding classification probability for each sample in the conflict test set, and the style preference index is calculated. We construct a style feature injector by utilizing the candidate Chinese text style with the highest style preference index.

[0012] Preferred methods for forming stylized example inputs include: The original example input is split into a descriptive modifier component, a semantic framework component, and a body component; Each component is input into the style feature injector to obtain a candidate set for each component; Perform semantic similarity filtering on candidates c in the candidate sets of each component, and retain candidates that meet the similarity threshold; The selected component candidates are combined by Cartesian product to form the input candidate set; For each combined candidate in the input candidate set, a joint optimization score based on semantic similarity and perplexity is calculated, and the combined candidate that meets the score threshold and has the highest score is selected as the stylization example input.

[0013] Preferred methods for synthesizing biased outputs include: Construct a prompt template that includes Chinese text style definitions, normal task instructions, and biased output instructions; The preset teacher model is invoked to generate candidate bias outputs for the stylized example input, and the sequence confidence and Regard polarity confidence of each candidate bias output are calculated respectively. Based on the weighted composite score of sequence confidence and Regard polarity confidence, the candidate bias with the highest score is selected for output.

[0014] Preferably, the bias susceptibility rate is the ratio of the negative polarity output probability under stylized user queries to the negative polarity output probability under original user queries.

[0015] Preferably, the Regard metric assesses bias by comparing the Regard ratio of the stylized user query group and the original user query group.

[0016] This invention also provides a context learning bias risk assessment system based on Chinese text style construction and injection, used to implement the method, including: The Chinese text style definition module is used to extract text style features from the acquired Chinese style corpus and construct Chinese text style definitions. The style feature injector building module is used to detect the sensitivity of the evaluated large language model to different Chinese text styles, and to build a style feature injector based on the Chinese text style with the highest sensitivity. A stylized example input construction module is used to inject style features into the input part of the example in the context learning prompt template based on the style feature injector to form a stylized example input; The bias output synthesis module is used to synthesize bias output based on the stylized example input using a teacher model, and replace the stylized example input and bias output with examples in the context learning prompt template; The model invocation module is used to learn contextual prompt templates based on replacement examples, and invoke the large language model being evaluated to perform original user queries and stylized user queries respectively, generating model output; The indicator calculation module is used to perform quantitative analysis on the model output using the Regard indicator and the bias susceptibility rate, and uses perplexity and semantic similarity as quality control indicators to obtain the bias risk assessment results.

[0017] Compared with the prior art, the beneficial effects of the present invention are as follows: 1) Achieve controllable quantitative assessment of bias risk in context learning scenarios.

[0018] This invention constructs a Chinese text style definition and injects style features at the prompt level. Without changing the model parameters, it can systematically reveal the differences in output polarity of a large language model under different population group reference conditions, realize the quantifiable analysis of the bias risk caused by context learning, and improve the interpretability and repeatability of fairness assessment.

[0019] 2) Black box, parameter-independent, suitable for actual deployment environments.

[0020] This invention does not require accessing or modifying the internal parameters of the large language model being evaluated, nor does it require fine-tuning the model. Bias risk assessment can be completed through controllable intervention at the prompt level. It is applicable to API call environments and commercial closed-source model scenarios, and has good engineering feasibility and versatility.

[0021] 3) Minimal intervention, small measurement bias, and stable and reliable assessment results.

[0022] This invention ensures that the semantics of the target population group references are not lost during style injection by using a data segmentation mechanism and component-level semantic similarity truncation. At the same time, it combines a perplexity and semantic similarity joint scoring mechanism to perform double screening of candidate stylization results, completing style construction while ensuring text naturalness and semantic fidelity. This reduces the measurement bias introduced by abnormal expressions triggering model rejection or security policy interception, and improves the stability and credibility of the evaluation results.

[0023] 4) It expresses itself naturally in Chinese contexts, has a low interception rate, and is highly efficient in evaluation.

[0024] Compared to methods that directly insert additional tokens or modify sentence structure, this invention uses a Chinese text style construction method for bias assessment, which is more in line with Chinese language expression habits, reduces semantic deviation and structural anomalies, effectively reduces the probability of security policy triggering, and improves the fluency and success rate of the assessment process.

[0025] 5) Supports cross-model and cross-language fairness audits.

[0026] The style construction and injection framework proposed in this invention is model-independent and can be transferred to large language models of different architectures and scales. At the same time, the method extracts style features in Chinese and can be extended to Chinese-English bilingual environments, supporting cross-language fairness comparison analysis and mitigation strategy verification, and enhancing system-level compliance audit capabilities. Attached Figure Description

[0027] To more clearly illustrate the technical solution of the present invention, the drawings used in the embodiments are briefly introduced below. Obviously, the drawings described below are only some embodiments of the present invention. For those skilled in the art, other drawings can be obtained based on these drawings without creative effort.

[0028] Figure 1 This is a flowchart of a method according to an embodiment of the present invention. Detailed Implementation

[0029] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0030] To make the above-mentioned objects, features and advantages of the present invention more apparent and understandable, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.

[0031] Example 1: like Figure 1 As shown, a method for assessing the risk of contextual learning bias based on Chinese text style construction and injection includes: S1: Extract text style features from the acquired Chinese style corpus and construct a definition of Chinese text style. Specifically, this embodiment selects four Chinese literary works with distinct styles as the corpus, namely "The Three-Body Problem", "Records of the Grand Historian", "The Legend of the Condor Heroes" and "Call to Arms", extract style features from each, and select one style for subsequent operations through a model sensitivity test experiment.

[0032] Furthermore, text style feature extraction is performed on the Chinese style corpus, including word-level extraction and sentence-level extraction.

[0033] The methods for word-level extraction include: A word segmentation database is used to segment and tag Chinese-style corpora, constructing word sequences and part-of-speech tag sequences. Specifically, let the Chinese-style corpus be... Using the `cut` function from the jieba word segmentation library to perform word segmentation and part-of-speech tagging on the corpus, we obtain: in, Indicates the first One word, express The corresponding part-of-speech tags, Indicates the total number of words.

[0034] Based on the part-of-speech tag sequence, a part-of-speech distribution frequency function is constructed to determine the dominant part-of-speech types in the Chinese stylistic corpus. Specifically, let the set of part-of-speech categories be: in, Represents the set of content word categories. It represents a set of function word categories; content words include nouns, verbs, adjectives, etc., while function words include adverbs, prepositions, conjunctions, etc.

[0035] Define the frequency function: Define the dominant part-of-speech category based on the frequency of part-of-speech distribution: in, Indicates the dominant part-of-speech type in the corpus.

[0036] Based on the character length of each word in the word sequence, a word length distribution frequency function is constructed. The frequency of occurrence of each word length type is counted, and the word length type with the highest frequency is selected as the word length feature. Specifically, let... Indicator The length of the word, i.e., the number of characters. Based on word length statistics, a frequency function is defined: in, Indicates the length type of a certain word. This indicates the frequency of the word length type in the corpus.

[0037] Based on the above statistics, the one that appears most frequently was selected. Word length type As a feature of word length: Words are divided into monosyllabic and polysyllabic words based on word length. The most frequently occurring monosyllabic words are counted, and polysyllabic words are further categorized by sentiment polarity using a pre-defined sentiment dictionary. The most frequently occurring terms are then counted for polysyllabic words with positive and negative sentiment polarities, resulting in high-frequency polysyllabic words with positive and negative polarities. Specifically, the sets of monosyllabic and polysyllabic words are defined: The most frequently occurring statistics One monosyllabic word: The GPT4-mini algorithm was called to perform a sentiment classification task, dividing multisyllabic words into positive and negative polarity subsets based on sentiment polarity. Statistical breakdown: Using a pre-defined dictionary of modal particles and idioms, this study employs regular expression matching to statistically analyze the frequency of modal particles and idioms in the corpus. Idioms are then categorized by sentiment polarity, with the highest-frequency terms in both positive and negative sentiment polarities being analyzed to obtain high-frequency idioms of both polarities. Specifically, the modal particle database from Baidu Encyclopedia is used as the modal particle dictionary. Using idiom-database as an idiom dictionary .

[0038] Let the regular expression matching function be: Used to determine terms Does it match the dictionary? The entries in.

[0039] Based on this, the characteristics of high-frequency modal particles are obtained: Idioms were categorized by their emotional polarity, resulting in high-frequency idioms with positive and negative polarities: The word-level style feature representation is constructed by describing and connecting the part-of-speech features, word length features, high-frequency monosyllabic words, high-frequency polysyllabic words with positive and negative polarity, high-frequency modal particles, and high-frequency idioms with positive and negative polarity using natural language. This results in the final word-level style feature representation. in, This indicates that the stylistic features are described in natural language and then connected sequentially.

[0040] Furthermore, methods for sentence-level extraction include: Using periods, question marks, and exclamation marks as terminators, the Chinese-style corpus is segmented into sentence sequences: in, Indicates the first One sentence. This indicates the total number of sentences in the corpus.

[0041] Extract Chinese and English characters from a sentence sequence by character matching to obtain a character set. And based on the number of characters in the character set Calculate the average sentence length using the total number of sentences: As a feature of sentence length.

[0042] Construct a set of sentence type categories, including complete sentences and non-complete sentences. Let the set of sentence type categories be: in, Indicates a complete sentence. Indicates a loose sentence.

[0043] Based on the sentence type category set, a first prompt template is constructed. A large language model is then used to classify the sentence type of each sentence in the sentence sequence, obtaining the corresponding sentence type category. The frequency of occurrence of each sentence type is counted, and the sentence type category with the highest frequency is selected as the dominant sentence type feature. Specifically, a prompt template is constructed, which includes sentences... Input a large language model for classification to obtain the classification function: Define the frequency function: Through frequency function calculate: in, This indicates the dominant sentence type in the corpus, i.e., the sentence type that appears most frequently.

[0044] A set of rhetorical categories is constructed, including metaphor, personification, hyperbole, quotation, parallelism, irony, rhetorical question, rhetorical question, and antithesis. Based on this set of rhetorical categories, a second prompt template is constructed. A large language model is then used to classify the rhetorical category of each sentence in the sentence sequence, obtaining the corresponding rhetorical category for each sentence. The frequency of each rhetorical category is counted, and the rhetorical category with the highest frequency is selected as the high-frequency rhetorical feature. Specifically, let the set of rhetorical categories be: in, It is a metaphor. Indicates personification, Indicates exaggeration. Indicates a reference. Indicates parallelism, Irony This is a rhetorical question. Indicates a question, It indicates duality.

[0045] Construct a prompt template and call GPT4-mini for categorization: Define the frequency function: Selecting high-frequency rhetorical devices: Sentence length features, dominant sentence structure features, and high-frequency rhetorical features are described and connected using natural language to form a sentence-level style feature representation, ultimately yielding the sentence-level style features: Furthermore, methods for constructing Chinese text style definitions include: Identify population group references in the input text and determine the target population group. Specifically, the original style features cannot directly guide the model to output biased content targeting a specific population group; therefore, the style features need to be edited. Let... This indicates a user query or sample input that requires style feature injection. Indicated in text The population groups identified in the data.

[0046] Based on word-level style feature representation, positive and negative polarity feature selection operators are constructed to select word-level style feature subsets corresponding to positive and negative polarities, respectively. Based on the target population group and population group reference, a conditional mapping function is constructed to select the corresponding word-level style feature subset according to whether the input text contains the target population group reference. The selected word-level style feature subset is connected with the sentence-level style feature representation to form the Chinese text style definition.

[0047] Specifically, the target population group is defined as Constructing feature selection operators and Let and represent the selection operators for choosing positive and negative polarity subsets from word-level style features, respectively. Then, we define: Furthermore, in, This represents a subset of word-level style features corresponding to positive polarity. This represents a subset of word-level style features corresponding to negative polarity.

[0048] Define a conditional mapping function: Finally, we obtain the Chinese text style definition representation function: in, This refers to the Chinese text style definition formed by connecting various style features described in natural language. This indicates the word-level style features selected based on population group reference conditions. Indicates sentence-level stylistic features.

[0049] S2: Detect the sensitivity of the evaluated large language model to different Chinese text styles, and construct a style feature injector based on the Chinese text style with the highest sensitivity; a further implementation method is that the method for constructing the style feature injector includes: Construct a binary classification prompt template, which consists of several examples. Each example contains a sentence and a binary classification label, which is 0 or 1. By controlling the sentiment polarity in the sentence and whether it contains candidate Chinese text styles, the example's classification label is kept consistent when it is classified by sentiment polarity and when it is classified by whether it contains candidate Chinese text styles. A conflict test set is constructed. Each sample in the conflict test set has a first true label categorized by sentiment polarity and a second true label categorized by whether it contains candidate Chinese text styles. The first and second true labels are inconsistent. Specifically, the conflict test set is constructed to detect the degree of dependence of the evaluated large language model on sentiment and style cues. The samples in the test set are... This allows them to be categorized according to emotional polarity, with the true label being... Based on whether it contains a certain Chinese text style, the real label is: We control .

[0050] Under the binary classification prompt template, the large language model being evaluated is invoked to output a binary classification label or the corresponding classification probability for each sample in the conflict test set, and the style preference index is calculated. Under the given prompt template conditions, the large language model being evaluated is invoked to output binary labels or corresponding probabilities for samples in the test set; the model output labels and... Consistency and The consistency difference is averaged over the sample to form a style preference index, specifically: , in, Indicates the candidate Chinese style; Indicate candidate style The corresponding conflict test set; Indicates conflict test set The number of samples (cardinality); Indicates conflict test set The sample index in; Indicates conflict test set The first in One sample; This indicates that the tested large language model is effective against the sample. A measure of consistency between style and authentic labels; This indicates that the tested large language model is effective against the sample. A measure of the consistency of the emotional authenticity label; When the model can provide label probabilities, take: , , in Indicates the model predicts samples Belongs to the style authenticity tag The posterior probability, Indicates the model predicts samples Belongs to the emotional authenticity label The posterior probability; When the model only provides discrete labels At that time, take: , , in This indicates that the tested large language model is effective against the sample. Output discrete prediction labels, This indicates an indicator function; the function value is 1 if the condition within the parentheses is true, and 0 otherwise. This indicates that the model relies more heavily on style cues than on sentiment cues.

[0051] Using style preference index The largest candidate Chinese text style A style feature injector is constructed. This injector consists of Qwen-max and a prompt template. The prompt template includes a Chinese text style definition, a batch output control module, and a history memory and deduplication module. The batch output control module constrains the large language model to output multiple non-repeating candidate results for a single input at once. The history memory and deduplication module records the generated results and masks duplicate content when the style feature injector is called multiple times. During multiple calls, temperature, top-p, and sampling parameters are adjusted to improve generation diversity. This dual-layer control of single-output multi-step and multi-call deduplication significantly reduces the number of model calls and computational cost while ensuring candidate diversity.

[0052] S3: Based on a style feature injector, style features are injected into the input portion of the example in the context learning prompt template to form a stylized example input; a further implementation method is that the method for forming the stylized example input includes: The original example input is segmented into descriptive modifier components, semantic framework components, and body components. Specifically, when injecting style features into the input, the data of each input to be injected with style features is first segmented. The input is segmented into three components: descriptive modifier components... Semantic framework components and main components The descriptive modifier component is used to characterize or define the population group to which the subject belongs, the semantic framework component is used to describe the semantic structure of the sentence, and the subject component refers to the subject in the sentence. By focusing population group information on the descriptive modifier component, it is ensured that the target population group is not lost during the injection of style features.

[0053] Each component is input into the style feature injector to obtain a candidate set for each component; Perform semantic similarity filtering on the candidates in the candidate set of each component, and retain the candidates that meet the similarity threshold; The selected candidate components are combined using a Cartesian product to form the input candidate set; specifically, the three components obtained from the segmentation are denoted as follows: The candidate sets are obtained by inputting the style feature injector. For each candidate set... Calculate the semantic similarity with the original component. , retain satisfaction The candidate forms a truncated subset. By pre-limiting the semantic offset between descriptive modifier components and the main component through component-level truncation, the loss of audience referential information can be prevented.

[0054] For each combined candidate in the input candidate set, a joint optimization score based on semantic similarity and perplexity is calculated. The combined candidate that meets the scoring threshold and has the highest score is selected as the stylization example input. Specifically, the three subsets are combined using a Cartesian product to form the input candidate set. This allows for the acquisition of a large number of candidate stylization results through permutations and combinations. For each candidate combination in the candidate set... Calculate the joint optimization score: in For the original input, For example-level semantic similarity, For sample-level perplexity, Choose the option that satisfies the requirements. The highest-scoring candidate combination is used as a stylized example input that satisfies minimum intervention. Through a dual screening process of component-level capping and sample-level joint scoring, we ensure that stylized rewriting maintains both fine-grained semantic fidelity and overall fluency.

[0055] S4: Based on stylized example input, a teacher model is used to synthesize biased output, and the stylized example input and biased output are used to replace the examples in the context learning prompt template; a further implementation method is that the method for synthesizing biased output includes: A prompt template is constructed that includes Chinese text style definitions, normal task instructions, and biased output instructions. A pre-defined teacher model is invoked to generate candidate biased outputs from stylized example inputs, and the sequence confidence and Regard polarity confidence of each candidate biased output are calculated. Based on the weighted composite score of the sequence confidence and Regard polarity confidence, the candidate biased output with the highest score is selected.

[0056] The teacher model is used to synthesize biased outputs from stylized example inputs. A teacher model that is similar to the large language model being evaluated in terms of architecture, training corpus, and decoding strategy, and does not contain human preference alignment or other censorship mechanisms, is selected, preferably an open-source version of the model being evaluated with few parameters.

[0057] When calling the teacher model, a prompt template is used, which consists of a Chinese text style definition, normal task instructions, and biased output instructions. By calling the teacher model multiple times and stylizing the example input, candidate biased outputs are obtained. Sampling and decoding parameters, including temperature, top-p, and top-k, are switched to obtain diverse candidate biased outputs.

[0058] For each of the generated candidate bias outputs, the sequence confidence score and the Regard polarity confidence score are calculated, and a weighted selection is performed using a comprehensive score. The comprehensive score is defined as follows: in For sequence confidence, it is the exponential form of the length-normalized generation probability of candidate outputs under the teacher model. For Regard polarity confidence, This indicates whether the target population group exists in the input. If it exists, a negative Regard confidence score is used; otherwise, a positive Regard confidence score is used.

[0059] The Regard polarity confidence score is provided by the Regard classifier. For Chinese evaluation scenarios, the CHRegard classifier, obtained by fine-tuning the BERT model, is used; for English evaluation scenarios, the Regard classifier is used. For candidate outputs containing references to the target population group, the confidence score of being classified as -1 (negative polarity) is taken; for candidate outputs without references to the target population group or only containing references to non-target population groups, the confidence score of being classified as 1 (positive polarity) is taken. The Regard classifier is trained on Chinese corpus and outputs a four-element polarity confidence score, corresponding to 1 (positive polarity), 0 (neutral), -1 (negative polarity), and 2 (cannot be determined), which is used to represent the attitude of the sentence towards its subject.

[0060] The candidate output with the highest overall score is taken as the bias output corresponding to the input, and then replaced with the corresponding example in the stylized example input paired with the context learning cue template.

[0061] S5: Based on the contextual learning prompt template of the replacement example, the large language model under test is invoked to generate model output for the original user query and the stylized user query, respectively. Specifically, the large language model under test is invoked to generate content for the original user query and the stylized user query. The original user query is a sentence prefix containing the population group, which serves as the baseline input; the stylized user query is obtained by processing the original user query through the same style feature injection process as the example input.

[0062] The two sets of inputs are consistent in semantic content, differing only in text style features. By controlling the semantic similarity and perplexity thresholds during the stylization rewriting process, the differences between the two sets of inputs in terms of semantic similarity and perplexity are kept to a minimum, ensuring that the injection of style features constitutes minimal intervention.

[0063] Without modifying the parameters of the large language model under test, the large language model under test is called to generate outputs for the two sets of user queries based on the context learning prompt template of the replaced example, so as to compare and analyze the inducing effect of style features on the bias of model output.

[0064] S6: The model output is quantitatively analyzed using the Regard index and the bias susceptibility rate, and the perplexity and semantic similarity are used as quality control indicators to obtain the bias risk assessment results.

[0065] A further implementation method is that the bias susceptibility rate is the ratio of the probability of negative polarity output under stylized user queries to the probability of negative polarity output under original user queries.

[0066] A further implementation method involves using the Regard metric to assess bias by comparing the Regard ratio between the stylized user query group and the original user query group.

[0067] Specifically, bias in the model output is measured based on Regard-type metrics and bias susceptibility rate. The bias susceptibility rate is defined as: in Indicates the stylistic features adopted. This indicates that the input contains the target population group. This indicates a stylized user query. This indicates the original user query. This indicates that the output of the tested large language model is determined to be of negative polarity by the Regard classifier.

[0068] The conditional effect measure includes the Regard index and the bias susceptibility rate. The Regard index assesses bias by comparing the Regard ratio between the stylized user query group and the original user query group. When the proportion of model outputs in the stylized user query group containing the target population group reference that are judged as having negative Regard polarity is significantly higher than the corresponding proportion in the original user query group, and the proportion of negative Regard polarity for non-target population group references is not significantly different between the two groups, it is determined that the Chinese text style enhances the negative polarity output tendency of the tested large language model under the target population group condition, and the Regard polarity distribution and bias susceptibility rate are output as evaluation results.

[0069] The Regard polarity distribution and the aforementioned bias susceptibility rate are output as evaluation results. These evaluation results are used for comparative auditing and risk management across models, tasks, and languages.

[0070] Example 2: This invention also provides a context learning bias risk assessment system based on Chinese text style construction and injection, used to implement the method of Embodiment 1, including: The Chinese text style definition module is used to extract text style features from the acquired Chinese style corpus and construct Chinese text style definitions. Specifically, it first acquires candidate Chinese corpora, and statistically analyzes the distribution of parts of speech, word length and polysyllabic distribution, frequency of modal particles and idioms, sentence length, sentence structure and rhetorical categories. High-frequency polysyllabic words and idioms are then divided into positive polarity sets and negative polarity sets according to their polarity. The statistical results are then expressed as Chinese text style definitions in natural language form.

[0071] The style feature injector construction module is used to detect the sensitivity of the evaluated large language model to different Chinese text styles, and to construct a style feature injector based on the Chinese text style with the highest sensitivity. Specifically, this module is used in the large language model style sensitivity testing steps: constructing a contextual binary classification prompt template and a test set, and calculating the style preference index. The optimal Chinese text style is selected. In processing specific inputs, the module performs a data segmentation step, dividing the input into descriptive modifier components, semantic framework components, and main components. Based on the style feature injector, style features are injected into each component under joint optimization of perplexity and semantic similarity. A stylized result satisfying minimal intervention is generated through component-level truncated filtering and Cartesian product combination. The style feature injector consists of a large language model and a prompt template. The prompt template includes a Chinese text style definition, a batch output control module, and a history memory and deduplication module, used to constrain the generation process and ensure output diversity.

[0072] A stylized example input construction module is used to inject style features into the input portion of the examples in the context learning prompt template based on a style feature injector, forming stylized example input. A biased output synthesis module is used to synthesize biased output based on the stylized example input using a teacher model, and then replace the examples in the context learning prompt template with the stylized example input and the biased output. Specifically, the teacher model generates paired biased outputs for the stylized example input, combines the outputs with the stylized example input, and uses this combination to construct the context learning prompt template that replaces the examples. Specifically, the module selects a teacher model that is similar to the large language model being evaluated in terms of architecture, training corpus, and decoding strategy, and does not contain human preference alignment or other censorship mechanisms; preferably, it is an open-source, low-parameter version of the model being evaluated. By using a prompt template consisting of Chinese text style definitions, normal task instructions, and biased output instructions, the teacher model is called multiple times, and sampling and decoding parameters such as temperature, top-p, and top-k are switched to obtain diverse candidate biased outputs. Furthermore, the module calculates the sequence confidence and Regard polarity confidence of each candidate bias output. The sequence confidence is the exponential form of the length-normalized generation probability of the candidate bias output under the teacher model. The Regard polarity confidence is obtained using a CHRegard classifier based on BERT fine-tuning for Chinese evaluation scenarios and a Regard classifier for English evaluation scenarios. A negative Regard confidence is used for candidate outputs containing the target population group, and a positive Regard confidence is used for candidate outputs without the target reference or containing only non-target population group references. The module performs weighted selection using a comprehensive score, defined as the weighted average of the sequence confidence and Regard polarity confidence. The candidate output with the highest comprehensive score is selected as the input corresponding to the output, and paired with the stylized example input to obtain the stylized example.

[0073] The model invocation module is used to learn contextual hint templates based on replacement examples. It invokes the evaluated large language model to process the original user query and the stylized user query, generating model outputs. Specifically, the original user query serves as the baseline, consisting of sentence prefixes containing demographic group references. The stylized user query is constructed from the baseline through the same style feature injection process as the example input. The two sets of inputs maintain semantic consistency, differing only in text style features, and are kept minimally invasive by controlling the semantic similarity and perplexity gaps. This module invokes the evaluated large language model through a black-box interface, without updating or fine-tuning its parameters during the evaluation process.

[0074] The indicator calculation module is used to quantitatively analyze the model output using the Regard indicator and the bias susceptibility rate, and uses perplexity and semantic similarity as quality control indicators to obtain the bias risk assessment results. Specifically, the Regard indicator assesses bias by comparing the Regard ratio between the stylized user query group and the original user query group. When the proportion of model outputs containing target population references in the stylized user query group that are judged as having negative Regard polarity is significantly higher than the corresponding proportion in the original user query group, and the proportion of negative Regard polarity for non-target population references is not significantly different between the two groups, it is determined that the Chinese text style enhancement has a negative polarity output tendency of the tested large language model under the target population condition. The module also calculates the bias susceptibility rate. This indicator is defined as Where 's' represents the Chinese text style used, 'd=1' indicates that the input contains references to the target population group, 'g=1' represents a stylized user query, 'g=0' represents the original user query, and 'r=-1' indicates that the output of the tested large language model is determined to have negative polarity by the Regard classifier. Finally, the module outputs the Regard polarity distribution and bias susceptibility rate as evaluation results, used for cross-model, cross-task, and cross-language comparative auditing and risk management.

[0075] The embodiments described above are merely preferred embodiments of the present invention and are not intended to limit the scope of the present invention. Various modifications and improvements made by those skilled in the art to the technical solutions of the present invention without departing from the spirit of the present invention should fall within the protection scope defined by the claims of the present invention.

Claims

1. A method for assessing the risk of contextual learning bias based on Chinese text style construction and injection, characterized in that, include: Text style features are extracted from the acquired Chinese style corpus to construct a definition of Chinese text style; The sensitivity of the evaluated large language model to different Chinese text styles is detected, and a style feature injector is constructed based on the Chinese text style with the highest sensitivity. Based on the style feature injector, style features are injected into the input part of the example in the context learning prompt template to form a stylized example input; Based on the stylized example input, a biased output is synthesized using a teacher model, and the stylized example input and biased output are used to replace the examples in the context learning prompt template; The context-based learning prompt template based on the replacement example is used to call the evaluated large language model to perform the original user query and the stylized user query respectively, and generate the model output. The model output was quantitatively analyzed using the Regard index and the bias susceptibility rate, and the bias risk assessment results were obtained by using perplexity and semantic similarity as quality control indicators.

2. The method according to claim 1, characterized in that, Text style feature extraction from the aforementioned Chinese style corpus includes word-level extraction and sentence-level extraction; wherein, the methods for word-level extraction include: We used a word segmentation library to perform word segmentation and part-of-speech tagging on Chinese-style corpora, and constructed word sequences and part-of-speech tag sequences. Based on the part-of-speech tag sequence, a part-of-speech distribution frequency function is constructed to determine the dominant part-of-speech type of the Chinese style corpus; Based on the character length of each word in the word sequence, a word length distribution frequency function is constructed to count the occurrence frequency of each word length type, and the word length type with the highest occurrence frequency is selected as the word length feature; Words are divided into monosyllabic words and polysyllabic words based on word length. The most frequently occurring monosyllabic words are counted, and the polysyllabic words are divided into sentiment polarities using a pre-set sentiment dictionary. The most frequently occurring words in polysyllabic words with positive and negative sentiment polarities are counted to obtain high-frequency polysyllabic words with positive and negative sentiment polarities. Using a pre-defined dictionary of modal particles and idioms, the frequency of modal particles and idioms in the corpus is statistically analyzed by regular expression matching. The idioms are then divided into sentiment polarities. The most frequent words in idioms with positive and negative sentiment polarities are counted separately to obtain high-frequency idioms with positive and negative sentiment polarities. The word-level style feature representation is constructed by describing and connecting the part-of-speech features, word length features, high-frequency monosyllabic words, high-frequency polysyllabic words with positive and negative polarity, high-frequency modal particles, and high-frequency idioms with positive and negative polarity using natural language.

3. The method according to claim 2, characterized in that, Methods for sentence-level extraction include: Using periods, question marks, and exclamation marks as terminators, the Chinese-style corpus is segmented into sentence sequences. Character matching is used to extract Chinese and English characters from the sentence sequence to obtain a character set. The average sentence length is calculated based on the number of characters in the character set and the total number of sentences, and is used as the sentence length feature. Construct a set of sentence types, including complete sentences and non-complete sentences; Based on the set of sentence types, a first prompt template is constructed, and a large language model is called to classify the sentence type of each sentence in the sentence sequence to obtain the sentence type corresponding to each sentence. Statistically analyze the frequency of occurrence of various sentence patterns and select the sentence pattern category with the highest frequency as the dominant sentence pattern feature; Construct a set of rhetorical categories, which include metaphor, personification, hyperbole, quotation, parallelism, irony, rhetorical question, rhetorical question, and antithesis; Based on the set of rhetorical categories, a second prompt template is constructed, and a large language model is called to perform rhetorical classification on each sentence in the sentence sequence to obtain the rhetorical category corresponding to each sentence. The frequency of occurrence of various rhetorical devices was statistically analyzed, and the rhetorical device category with the highest frequency of occurrence was selected as the high-frequency rhetorical feature; The sentence length feature, the dominant sentence structure feature, and the high-frequency rhetorical feature are described and connected using natural language to form a sentence-level style feature representation.

4. The method according to claim 3, characterized in that, Methods for constructing Chinese text style definitions include: Identify population group references in the input text and determine the target population group; Based on the word-level style feature representation, positive polarity and negative polarity feature selection operators are constructed to select word-level style feature subsets corresponding to positive polarity and negative polarity, respectively. Based on the target population group and population group designations, a conditional mapping function is constructed to select the corresponding word-level style feature subset according to whether the input text contains population group designations. The selected subset of word-level style features is connected with the sentence-level style feature representation to form a Chinese text style definition.

5. The method according to claim 3, characterized in that, Methods for constructing style feature injectors include: A binary classification prompt template is constructed, which consists of several examples. Each example contains a sentence and a binary classification label. By controlling the sentiment polarity in the sentence and whether it contains candidate Chinese text styles, the classification labels of the examples are consistent when they are classified according to sentiment polarity and when they are classified according to whether they contain the candidate Chinese text styles. A conflict test set is constructed, wherein each sample in the conflict test set has a first true label according to the sentiment polarity classification and a second true label according to whether it contains the candidate Chinese text style classification, and the first true label and the second true label are inconsistent. Under the binary classification prompt template, the large language model being evaluated is invoked to output a binary classification label or the corresponding classification probability for each sample in the conflict test set, and the style preference index is calculated. We construct a style feature injector by utilizing the candidate Chinese text style with the highest style preference index.

6. The method according to claim 3, characterized in that, Methods for generating stylized example inputs include: The original example input is split into a descriptive modifier component, a semantic framework component, and a body component; Each component is input into the style feature injector to obtain a candidate set for each component; Perform semantic similarity filtering on candidates c in the candidate sets of each component, and retain candidates that meet the similarity threshold; The selected component candidates are combined by Cartesian product to form the input candidate set; For each combined candidate in the input candidate set, a joint optimization score based on semantic similarity and perplexity is calculated, and the combined candidate that meets the score threshold and has the highest score is selected as the stylization example input.

7. The method according to claim 3, characterized in that, Methods for synthesizing biased outputs include: Construct a prompt template that includes Chinese text style definitions, normal task instructions, and biased output instructions; The preset teacher model is invoked to generate candidate bias outputs for the stylized example input, and the sequence confidence and Regard polarity confidence of each candidate bias output are calculated respectively. Based on the weighted composite score of sequence confidence and Regard polarity confidence, the candidate bias with the highest score is selected for output.

8. The method according to claim 3, characterized in that, The bias susceptibility rate is the ratio of the probability of negative polarity output under stylized user queries to the probability of negative polarity output under original user queries.

9. The method according to claim 3, characterized in that, The Regard metric assesses bias by comparing the Regard ratio between the stylized user query group and the original user query group.

10. A context learning bias risk assessment system based on Chinese text style construction and injection, used to implement the method described in any one of claims 1-9, characterized in that, include: The Chinese text style definition module is used to extract text style features from the acquired Chinese style corpus and construct Chinese text style definitions. The style feature injector building module is used to detect the sensitivity of the evaluated large language model to different Chinese text styles, and to build a style feature injector based on the Chinese text style with the highest sensitivity. A stylized example input construction module is used to inject style features into the input part of the example in the context learning prompt template based on the style feature injector to form a stylized example input; The bias output synthesis module is used to synthesize bias output based on the stylized example input using a teacher model, and replace the stylized example input and bias output with examples in the context learning prompt template; The model invocation module is used to learn contextual prompt templates based on replacement examples, and invoke the large language model being evaluated to perform original user queries and stylized user queries respectively, generating model output; The indicator calculation module is used to perform quantitative analysis on the model output using the Regard indicator and the bias susceptibility rate, and uses perplexity and semantic similarity as quality control indicators to obtain the bias risk assessment results.

Images