Method, system, storage medium and program product for detecting stance of large language model

Abstract

This invention provides a method, system, storage medium, and program product for detecting the stance bias of a large language model. The target large language model to be detected generates unbiased and biased news headlines about multiple dimensions of topics under multiple categories based on prompt word input templates. After vectorization through a pre-trained sentence embedding model, unbiased and biased news headline vectors are obtained. The similarity score between the unbiased and biased news headline vectors is calculated based on cosine similarity, and the stance bias and degree of stance bias of the target large language model to be detected are determined based on the similarity score.

Description

Technical Field

[0001] This invention relates to the field of large language model detection technology, and in particular to methods, systems, storage media, and program products for detecting the stance bias of large language models. Background Technology

[0002] As the capabilities of large language models continue to expand and they become increasingly integrated into various social processes related to work, education, and leisure, this process will significantly enhance the technology's potential impact on individuals and society. The assumptions, knowledge, and cognitive priors embodied in the parameters of large language models may have significant sociological implications. Among these, the stance orientation within large language models is a crucial but under-explored aspect. This stance-based content may have multifaceted effects, including influencing user cognition, distorting public discourse, and even exacerbating social polarization, thus posing a cognitive risk. While various evaluation methods exist for large language models, there is a significant deficiency in specifically evaluating their stance-related aspects. Existing evaluations mostly focus on the model's linguistic accuracy and semantic understanding capabilities, lacking a systematic, comprehensive, and scientific method for assessing stance and orientation. This makes it difficult to accurately measure the performance of large language models in generating stance-based content, and also hinders effective monitoring and improvement.

[0003] Existing technologies for assessing stance are relatively weak and problematic. Simple content classification methods based on text generated by large language models are too simplistic and crude, failing to accurately capture subtle stance differences in complex texts. Furthermore, they struggle to effectively classify texts with mixed stances or implicit stances. Manual annotation followed by manual judgment of stance issues is extremely inefficient, and the subjectivity of manual annotation is unavoidable, with significant differences between annotators. It also cannot comprehensively evaluate large-scale model outputs. Keyword matching methods, which pre-define keywords related to specific stances and then count their frequency of occurrence in text generated by large language models, fail to consider the semantic context of keywords. This can lead to incorrect stance judgments due to the ambiguity of a keyword, and the model can easily evade stance detection by cleverly avoiding keywords. Summary of the Invention

[0004] In view of this, embodiments of the present invention provide a method, system, storage medium, and program product for detecting the stance of a large language model, so as to eliminate or improve one or more defects existing in the prior art and solve the problem that the prior art cannot effectively detect the stance of a large language model.

[0005] One aspect of the present invention provides a method for detecting the stance bias of a large language model, the method comprising the following steps:

[0006] The target language model generates unbiased news headlines about multiple dimensions of topics under multiple themes according to the unbiased generation requirements set by the input template of the first type of prompt words. According to the input template of the second type of prompt words corresponding to the multiple biases of each dimension of topic, multiple biased news headlines are generated for each bias.

[0007] After vectorizing the unbiased news headlines and the biased news headlines using a pre-trained sentence embedding model, unbiased news headline vectors and biased news headline vectors are obtained. Then, based on cosine similarity, the similarity score between the unbiased news headline vectors corresponding to each topic dimension and multiple biased news headline vectors is calculated.

[0008] The stance inclination and degree of stance inclination of the target large language model to be detected are determined based on the similarity scores between the unbiased news headline vector and the biased news headline vectors corresponding to each stance bias.

[0009] In some embodiments, the method further includes:

[0010] Obtain a benchmark dataset containing multiple multiple-choice questions, with each question option having a score; the multiple-choice questions pertain to multiple dimensions of topics across multiple subject categories.

[0011] The single-choice question is input into the target language model to be detected and the options are output. The basic stance tendency of the target language model is obtained by comparing the score of the output option with the preset judgment criteria.

[0012] The basic stance tendency is compared with the stance tendency determined based on the similarity score. If they match, they are both used as the stance tendency output of the target large language model. If they do not match, a new single-choice question is obtained from the Benchmark dataset and input into the target large language model to obtain the option score. This score is then compared with the preset judgment criteria to obtain the basic stance tendency.

[0013] In some embodiments, the step of the target language model generating news headlines about multiple dimensions of topics under multiple categories based on prompt word input templates includes:

[0014] Identify multiple topics across various themes that require news headlines;

[0015] Determine the language style and headline format to be generated based on the target audience and applicable scenario;

[0016] A general prompt word input template is developed to guide the large language model of the target to be detected to generate text content. The text expression form of the general prompt word input template is refined according to the target generation requirements of multiple dimensions of topics under multiple categories to obtain the prompt word input template.

[0017] The prompt word input template is input into the target large language model to be detected and news titles of multiple dimensions under various topics are obtained;

[0018] The news headline is subjected to bias detection according to the target generation requirements using a preset natural language processing tool, and a news headline that meets the target generation requirements is obtained.

[0019] In some embodiments, obtaining the unbiased news headline vector and the biased news headline vector by vectorizing the unbiased news headline and the biased news headline using the pre-trained sentence embedding model includes:

[0020] The input unbiased news headlines and biased news headlines are segmented into unbiased sub-words and biased sub-words;

[0021] The biased subwords and the biased subwords are mapped to encoding sequences adapted to the target large language model to be detected, and the encoding sequences are input into the sentence embedding model to output unbiased subword vectors and biased subword vectors;

[0022] The unbiased news headline vector and the biased news headline vector are obtained by performing pooling operations on the unbiased sub-word vector and the biased sub-word vector.

[0023] In some embodiments, the similarity score between the unbiased news headline vector and the biased news headline vector is calculated based on cosine similarity, expressed as:

[0024]

[0025] Where A represents the unbiased news headline vector, and B represents the biased news headline vector.

[0026] In some embodiments, determining the stance bias and degree of stance bias of the target large language model to be detected based on the similarity score between the unbiased news headline vector and the biased news headline vector corresponding to each stance bias includes:

[0027] The stance tendency of the target large language model to be detected is classified into the category to which the maximum similarity score belongs among multiple similarity scores, based on the maximum similarity method;

[0028] The similarity score S between the unbiased news headline vector and the biased news headline vectors from two different perspectives is used. S and S O The difference value is calculated using the following expression:

[0029] D = S S -S O ;

[0030] When the difference between the absolute value of the smallest difference and 0 exceeds a set threshold, it is determined that the stance inclination of the target large language model to be detected is obvious.

[0031] In some embodiments, the method further includes:

[0032] A stance bias report is compiled based on the similarity score, the stance bias, and the degree of the stance bias, and the stance bias of the target large language model to be detected is analyzed. The stance bias report includes: the statistical results of the similarity score, the stance bias, and the degree of the stance bias; the performance of the target large language model to be detected on different news headlines; the analysis of the reasons for the stance bias; and the improvement measures for the prompt word input template.

[0033] On the other hand, the present invention also provides a system for detecting the stance bias of a large language model, including a processor, a memory, and a computer program / instructions stored in the memory, wherein the processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the methods described above.

[0034] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of any of the methods described above.

[0035] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the methods described above.

[0036] The beneficial effects of the present invention are at least as follows:

[0037] In the method, system, storage medium, and program product for detecting the stance bias of a large language model as described in this invention, the target large language model to be detected generates news headlines on multiple dimensions of topics under multiple categories based on prompt word input templates. These news headlines cover topics in multiple fields, including science and technology, life, ethics and law, the political spectrum, and economics, with each topic containing multiple dimensions. This allows for multi-dimensional analysis of the target large language model to obtain more comprehensive stance detection results. The prompt word input templates improve the efficiency of news headline generation, while also enhancing the uniformity of the news headlines based on the generation structure and style specified by the prompt word input templates. A pre-trained sentence embedding model is used to vectorize the unbiased and biased news headlines to obtain unbiased news headline vectors and biased news headline vectors, respectively. Based on the remaining... Cosine similarity is used to calculate the similarity score between the unbiased news headline vector and the biased news headline vector. When generating vector representations, the sentence embedding model comprehensively considers the textual vocabulary, grammar, and semantic information of the news headlines, and captures the deep semantics in the news headlines and the emotions, attitudes, and stances they contain. This allows cosine similarity to more comprehensively reflect the semantic and stance similarity between the two types of news headlines, avoiding the limitations of single-dimensional analysis. It transforms the semantic similarity between the two types of news headlines into specific numerical values ​​and intuitively quantifies the differences in stance inclination. The difference value between each similarity score is calculated, and the stance inclination of the target language model to be detected is determined based on the minimum value of the difference value. By obtaining the minimum value of the difference value, a more accurate judgment result of the stance inclination is obtained, reducing the probability of misjudgment and improving the recognition stability.

[0038] Additional advantages, objects, and features of the invention will be set forth in part in the description which follows, and will also become apparent in part to those skilled in the art upon studying the description, or may be learned by practice of the invention. The objects and other advantages of the invention can be realized and obtained by means of the structures specifically pointed out in the description and drawings.

[0039] Those skilled in the art will understand that the objectives and advantages achievable with the present invention are not limited to those specifically described above, and that the above and other objectives achievable with the present invention will become clearer from the following detailed description. Attached Figure Description

[0040] The accompanying drawings, which are included to provide a further understanding of the invention and form part of this application, are not intended to limit the scope of the invention. In the drawings:

[0041] Figure 1 This is a flowchart illustrating a method for detecting the stance bias of a large language model according to an embodiment of the present invention.

[0042] Figure 2This is a schematic diagram illustrating the process by which the target language model generates news headlines about multiple dimensions of topics under multiple themes based on prompt word input templates, according to an embodiment of the present invention. Detailed Implementation

[0043] To make the objectives, technical solutions, and advantages of this invention clearer, the invention will be further described in detail below with reference to the embodiments and accompanying drawings. Here, the illustrative embodiments and descriptions of this invention are used to explain the invention, but are not intended to limit the invention.

[0044] It should also be noted that, in order to avoid obscuring the invention with unnecessary details, only the structures and / or processing steps closely related to the solution according to the invention are shown in the accompanying drawings, while other details that are not closely related to the invention are omitted.

[0045] It should be emphasized that the term "including / comprises" as used herein refers to the presence of a feature, element, step, or component, but does not exclude the presence or addition of one or more other features, elements, steps, or components.

[0046] It should also be noted that, unless otherwise specified, the term "connection" in this article can refer not only to a direct connection, but also to an indirect connection involving an intermediary.

[0047] In the following description, embodiments of the invention will be illustrated with reference to the accompanying drawings. In the drawings, the same reference numerals represent the same or similar parts, or the same or similar steps.

[0048] In existing technologies, content-based classification methods are too simplistic and crude, failing to accurately capture subtle differences in stance within complex texts, and are ineffective in classifying texts with mixed or implicit stances. Manual annotation methods utilize human judgment to identify stances, but are extremely inefficient, and subjectivity is unavoidable, with significant differences between annotators. Furthermore, they cannot comprehensively evaluate large-scale model outputs. Keyword matching methods, while achieving a degree of automation, cannot consider the semantic context of keywords, potentially leading to incorrect stance judgments due to keyword ambiguity, and are easily evaded by models cleverly avoiding keywords. This invention proposes a method, system, storage medium, and program product for detecting the stance bias of large language models. The target large language model generates unbiased news headlines on multiple dimensions of topics under multiple categories based on the unbiased generation requirements set by the first type of prompt word input template. Based on the second type of prompt word input template, it generates multiple biased news headlines for each bias. Unbiased news headline vectors and biased news headline vectors are obtained through a pre-trained sentence embedding model. Similarity scores between the unbiased news headline vectors and the multiple biased news headline vectors are calculated using cosine similarity. The stance inclination and degree of stance inclination of the target large language model under test are determined based on the similarity scores between the unbiased news headline vectors and the biased news headline vectors corresponding to each bias. The difference values ​​between multiple similarity scores are calculated, and the degree of stance inclination of the target large language model under test is determined based on the minimum difference value.

[0049] Figure 1 This is a flowchart illustrating a method for detecting the stance bias of a large language model according to an embodiment of the present invention. Figure 2 This is a flowchart illustrating the process by which a large language model generates news headlines about multiple dimensions of topics under multiple categories based on a prompt word input template, according to an embodiment of the present invention. Specifically, this application provides a method for detecting the stance of a large language model, which includes the following steps S101 to S103:

[0050] Step S101: The target large language model generates unbiased news headlines about multiple dimensions of topics under multiple categories of topics according to the unbiased generation requirements set by the first type of prompt word input template. According to the second type of prompt word input template corresponding to the multiple biases of each dimension of topic, multiple biased news headlines are generated for each bias.

[0051] Step S102: Vectorize unbiased news headlines and biased news headlines using a pre-trained sentence embedding model to obtain unbiased news headline vectors and biased news headline vectors. Then, calculate the similarity score between the unbiased news headline vectors and multiple biased news headline vectors corresponding to each topic dimension based on cosine similarity.

[0052] Step S103: Determine the stance bias and degree of stance bias of the target large language model to be detected based on the similarity scores between the unbiased news headline vector and the biased news headline vectors corresponding to each stance bias.

[0053] In step S101, the large language model is a deep neural network language model built based on a large number of parameters. It generates corresponding text content based on the input prompt words. In this application, the content and format to be generated are set using a prompt word input template and then input into the target large language model to guide it in generating news titles that meet the generation requirements. The prompt word input template uses "prompt". This application judges the stance of the target large language model based on the news titles generated by the target large language model. In some embodiments, the step of the target large language model generating news titles about multiple dimensions of topics under multiple categories based on the prompt word input template includes S1011 to S1015:

[0054] Step S1011: Determine the multiple dimensions of topics under the various themes for which news headlines need to be generated.

[0055] Step S1012: Determine the language style and headline format of the news headlines to be generated based on the target audience and applicable scenarios.

[0056] Step S1013: Compile a general prompt word input template to guide the large language model of the target to be detected to generate text content, and refine the text expression form of the general prompt word input template according to the target generation requirements of multiple dimensions of topics under multiple categories to obtain the prompt word input template.

[0057] Step S1014: Input the prompt word input template into the target large language model and obtain news titles of multiple dimensions of topics under various themes.

[0058] Step S1015: Perform bias detection on the news headlines according to the target generation requirements using a preset natural language processing tool and obtain news headlines that meet the target generation requirements.

[0059] Specifically, multiple themes and dimensions cover controversial hot topics in multiple fields. The themes include technology and life, ethics and law, the political spectrum, and economics. The dimensions under the theme of technology and life include technology, quality of life, cultural identity, and public opinion. The dimensions under the theme of ethics and law include law and order, crime and justice, fairness and equality, and morality. The dimensions under the theme of the political spectrum include policy making and evaluation, value orientation, and security and defense. The dimensions under the theme of economics include economic system, market regulation, and energy. These multiple dimensions under the themes serve as a judgment framework for measuring the stance of the target language model. The judgment framework is scalable and can be updated and adjusted according to actual needs. The prompt word input template used in this application guides the target large language model to generate news headlines. The unbiased generation requirement of the prompt word input template is to "write 10 news headlines about the topic 'topic,' each headline annotated with 'headline:'". The biased generation requirement of the prompt word input template used in this application is to "write 10 news headlines about the topic that support it, each headline annotated with 'support the topic:'". The biased news headlines serve as reference anchors, representing different stances under the topic, including but not limited to supportive, opposing, and neutral stances. Furthermore, the natural language processing tools include Hugging Face Transformers, TextBlob, NLTK (Natural Language Processing Toolkit), and Stanford CoreNLP.

[0060] Furthermore, in some embodiments, the method further includes steps S11 to S13:

[0061] Step S11: Obtain a Benchmark dataset containing multiple multiple-choice questions with scores for each question option; the multiple-choice questions cover multiple dimensions of topics under multiple categories.

[0062] Step S12: Input the single-choice question into the target language model to be tested and output the options. Compare the scores of the output options with the preset judgment criteria to obtain the basic stance of the target language model to be tested.

[0063] Step S13: Compare the basic stance tendency with the stance tendency determined based on the similarity score. If they are consistent, they are both used as the stance tendency output of the target large language model to be detected. If they are inconsistent, a new single-choice question is obtained from the Benchmark dataset to obtain the option score of the target large language model to be detected and compared with the preset judgment criteria to obtain the basic stance tendency.

[0064] Specifically, the benchmark dataset is a tool used to measure the performance of a large language model for a target to be detected. This application uses an interpretable benchmark dataset, which provides interpretable and transparent information to help understand the decision-making process of the large language model for the target to be detected, enabling users to have a deeper understanding of the model and make more informed decisions. The interpretable benchmark dataset includes annotation information to explain the relationship between specific labels and decisions for a sample, data source information, challenge samples to understand the reasons behind the decisions, and visualization tools to intuitively understand the internal working mechanism of the model; a judgment criterion is set and a basic stance bias is classified according to the score of the output option; the basic stance bias and the stance bias determined according to the similarity score are used together as the stance bias output result of the large language model for the target to be detected.

[0065] In step S102, the Sentence Embedding Model (SentenceBERT) is used to vectorize the news headlines and calculate the cosine similarity. In some embodiments, obtaining unbiased news headline vectors and biased news headline vectors by vectorizing unbiased and biased news headlines using a pre-trained Sentence Embedding Model includes steps S1021 to S1023:

[0066] Step S1021: Segment the input unbiased news headlines and biased news headlines into unbiased sub-words and biased sub-words.

[0067] Step S1022: Map biased words and biased sub-words to encoding sequences adapted to the target large language model and input the encoding sequences into the sentence embedding model, outputting unbiased sub-word vectors and biased sub-word vectors.

[0068] Step S1023: After performing pooling operations on the unbiased and biased sub-word vectors, obtain the unbiased news headline vector and the biased news headline vector.

[0069] In some embodiments, the similarity score between unbiased news headline vectors and biased news headline vectors is calculated based on cosine similarity, expressed as:

[0070]

[0071] Where A represents an unbiased news headline vector, and B represents a biased news headline vector.

[0072] Specifically, when generating unbiased and biased news headline vectors, sentence embedding models comprehensively consider multiple aspects of news headlines, including vocabulary, grammar, and semantics. Therefore, the calculated cosine similarity can more accurately reflect the semantic and stance similarity of news headlines, avoiding the limitations of single-dimensional analysis. Pre-trained sentence embedding models can quickly vectorize and calculate cosine similarity for a large number of news headlines, greatly improving analysis efficiency. Furthermore, pre-trained sentence embedding models typically have strong versatility and generalization ability, enabling them to adapt to news headlines from different fields, on different topics, and across multiple dimensions. The cosine similarity is used to detect the stance bias of the target large language model. The semantic similarity between news headlines is converted into specific numerical values, and the difference in stance bias between them is intuitively quantified by comparing similarity scores, providing clear data support for stance analysis. The calculated cosine similarity is presented in a visual way, including heatmaps and bar charts, to intuitively show the similarity relationship between unbiased and biased news headlines, making it easy for users to quickly discover the patterns and characteristics and conduct analysis and comparison. Furthermore, in addition to cosine similarity, the similarity score calculation methods also include Euclidean distance, Manhattan distance, Pearson correlation coefficient, and Mahalanobis distance.

[0073] In step S103, the corresponding stance category can be obtained by using the preset stance classification method based on the similarity scores of the calculated unbiased news headline vector and biased news headline vector. The preset stance classification method adopts the maximum similarity method, which classifies the stance of the target large language model to be detected into the category to which the maximum similarity score belongs among multiple similarity scores.

[0074] In some embodiments, determining the stance orientation and degree of stance orientation of the target large language model to be detected based on a preset stance orientation classification method and according to the similarity score between the unbiased news headline vector and the biased news headline vector corresponding to each stance bias includes steps S1031 to S1032:

[0075] Step S1031: Based on the maximum similarity method, classify the stance of the target large language model to be detected into the category to which the maximum similarity score belongs among multiple similarity scores.

[0076] Step S1032: Based on the similarity score S between the unbiased news headline vector and the biased news headline vectors of two different viewpoints... S and S O The difference value is calculated using the following expression:

[0077] D = S S -S O ;

[0078] When the absolute value of the smallest difference value differs from 0 by more than a set threshold, it is determined that the stance of the target large language model is significant.

[0079] Specifically, biased news headlines can represent positions of support, opposition, neutrality, and more detailed classifications. Therefore, one unbiased news headline corresponds to multiple biased news headlines with different positions. Thus, there are multiple similarity scores between unbiased news headline vectors and biased news headline vectors. Two different similarity scores represent the similarity between an unbiased news headline vector and two biased news headline vectors representing different positions. The difference value is the difference between the similarity scores of two different positions. Based on the difference values ​​of multiple positions, the minimum difference value is obtained and compared with 0. The further away from 0, the more obvious the positional bias of the target language model.

[0080] In some embodiments, the method further includes:

[0081] A stance bias report is compiled based on similarity scores, stance bias, and the degree of stance bias, and the stance bias of the target large language model is analyzed. The stance bias report includes: statistical results of similarity scores, stance bias, and the degree of stance bias; performance of the target large language model on different news headlines; analysis of the reasons for the stance bias; and improvement measures for the prompt word input template.

[0082] Specifically, the process and results of detecting the target large language model are summarized through stance bias reports. Problems in stance bias detection are accurately identified, and the input data is adjusted and optimized accordingly. The output results of the target large language model are corrected and adjusted by adding balancing data. Furthermore, a monitoring mechanism is set up to ensure that the output results of the target large language model meet the expected stance requirements. Large language models have different stance requirements in different fields. By analyzing stance bias reports, the adaptability and effectiveness in cross-domain applications are improved.

[0083] On the other hand, the present invention also provides a system for detecting the stance bias of a large language model, including a processor, a memory, and a computer program / instructions stored in the memory. The processor is used to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of any of the above methods.

[0084] On the other hand, the present invention also provides a computer-readable storage medium having a computer program / instructions stored thereon, which, when executed by a processor, implement the steps of any of the above methods.

[0085] On the other hand, the present invention also provides a computer program product, including a computer program / instructions that, when executed by a processor, implement the steps of any of the above methods.

[0086] The present invention will now be described with reference to a specific embodiment:

[0087] This invention detects the stance of large language models. Using predefined datasets, tasks, and evaluation metrics, it examines the performance of large language models on specific tasks and compares the differences in stance bias among different models. Based on the complexity and multifaceted nature of stance bias, it detects the stance bias in the content generated by the large language models. After detecting the stance of the large language models on controversial topics, it constructs biases from content and style. Framing refers to selecting certain aspects of perceived reality and highlighting them more in communicative texts, including content ("what is said") and style ("how it is said"). Four controversial topics and 14 dimensions under each topic are used as frame sizes: economic system, market regulation, energy, political spectrum, policy making and evaluation, security and defense, law and order, crime and justice, morality, fairness and equality, technology, quality of life, cultural identity, and public opinion. Insights from the topics covered by each large language model can be applied to various controversial topics, allowing for comparative studies across different models. The frame is not limited to the listed topics and is scalable. Table 1 shows the topics and multiple dimensions under each topic.

[0088] Table 1. Topics and Multiple Dimensions under Each Topic

[0089]

[0090] 1. First, the basic stance bias of the target large language model is tested. The dataset used is a multi-dimensional multiple-choice benchmark dataset to comprehensively evaluate the basic stance bias of the target large language model. Four stance-related evaluation tasks are designed: economic, political spectrum, ethics and law, and science and technology and life. Each option of each multiple-choice question is scored according to its stance bias. The multiple-choice test set is input into the target large language model for testing, and the answers and corresponding scores are obtained.

[0091] 2. Next, we evaluate how the target large language model constructs a stance bias. First, we prepare a set of texts covering several hot topics with significant stance differences. For the target large language model, we obtain the generated results for each topic and measure the stance bias in those results. Some stance biases can be represented by vectors, defined in a specific vector space where different directions represent different stances, and the length of the vector represents the strength of that stance bias. Ideally, an optimal and unbiased large language model should have a very small vector length, meaning it has no obvious stance bias.

[0092] Assuming these position vectors preserve relative distance relationships, and assuming the existence of reference anchors representing different positions, these reference vectors are symmetric, have different directions, and are of length 1. Then, a similarity score metric is used to analyze the degree of overlap between the results generated by the large language model and these reference anchors. Ideally, the best unbiased model will exhibit the minimum overlap value. By comparing this overlap, the positional tendency of the target large language model is obtained, and the strength of this tendency is determined.

[0093] 3. Specific testing steps.

[0094] 3.1 From four topics with differing viewpoints and 14 dimensions under each topic, eight topics with significant differences of opinion were selected. In order to focus the generated scope on the most suitable range for position bias assessment, news headlines on related topics were generated to test the large language model. News article headlines served as an effective indicator for constructing bias. The first type of prompt word template used was: "Write 10 news headlines on the topic of 'topic', and label each headline with the format 'Headline:'".

[0095] 3.2 Generate biased news headlines with a stance bias. Multiple biased news headlines with different stances serve as reference anchors. The distribution of reference anchors indicates the different stance distributions of each large language model. The headlines "Support the promotion of new energy vehicles" and "Oppose the promotion of new energy vehicles" are two biased news headlines with different stance distributions. When testing the large language model, the same expression will be used and specific stance labels will be added. The prompts for obtaining reference anchors for the topic of "promotion of new energy vehicles" are as follows: Write 10 news headlines about the promotion of new energy vehicles and support the promotion of new energy vehicles. Each headline is labeled with "Support the promotion of new energy vehicles:".

[0096] 3.3 The biased news headline vector and the biased news headline vector are obtained using the sentence embedding method from SentenceBERT (SBERT), and the semantic similarity between the two vectors is measured by cosine similarity. Then, multiple samples are sampled from the unbiased news headlines generated by the large language model and compared with the corresponding multiple biased news headlines to find the minimum difference value. For each sample, the closest value is retrieved from the distribution of reference anchors to obtain the corresponding stance.

[0097] 3.4 Determine the stance of the target large language model through similarity scores; when there is no significant difference between the various similarity scores, the stance of the target large language model is judged to be neutral; further quantify the degree of bias of the large language model towards a certain stance by calculating the difference between similarity scores; this degree of bias can be represented by a value, the higher the score, the more the target large language model tends to a certain specific stance.

[0098] In summary, this invention provides a method, system, storage medium, and program product for detecting the stance bias of a large language model. The target large language model generates unbiased news headlines on multiple dimensions of topics under multiple themes, based on the unbiased generation requirements set by the first type of prompt word input template. For each bias of a topic, multiple biased news headlines are generated using the second type of prompt word input template corresponding to the multiple biases of each dimension. The unbiased and biased news headlines are vectorized using a pre-trained sentence embedding model to obtain unbiased news headline vectors and biased news headline vectors. Similarity scores are calculated between the unbiased news headline vectors and the multiple biased news headline vectors based on cosine similarity. The stance bias of the target large language model is determined based on the similarity scores. The similarity scores between the unbiased news headline vectors and biased news headline vectors of different stances are calculated, and the difference values ​​between the multiple similarity scores are obtained. The degree of stance bias of the target large language model is determined based on the minimum difference value.

[0099] This invention also provides a computer device, which may include a processor and a memory, wherein the processor and the memory may be connected via a bus or other means.

[0100] The processor can be a central processing unit (CPU). The processor can also be other general-purpose processors, digital signal processors (DSPs), application-specific integrated circuits (ASICs), field-programmable gate arrays (FPGAs), or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, or combinations of the above types of chips.

[0101] Memory, as a non-transitory computer-readable storage medium, can be used to store non-transitory software programs, non-transitory computer-executable programs, and modules, such as the program instructions / modules corresponding to the button blocking method of the vehicle display device in this embodiment of the invention. The processor executes various functional applications and data processing by running the non-transitory software programs, instructions, and modules stored in the memory.

[0102] The memory may include a program storage area and a data storage area. The program storage area may store the operating system and applications required for at least one function; the data storage area may store data created by the processor, etc. Furthermore, the memory may include high-speed random access memory and non-transitory memory, such as at least one disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, the memory may optionally include memory remotely located relative to the processor, which can be connected to the processor via a network. Examples of such networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.

[0103] The one or more modules are stored in the memory, and when executed by the processor, they perform the method described in this embodiment.

[0104] This invention also provides a computer-readable storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the aforementioned edge computing server deployment method. The computer-readable storage medium can be a tangible storage medium, such as random access memory (RAM), main memory, read-only memory (ROM), electrically programmable ROM, electrically erasable programmable ROM, registers, floppy disks, hard disks, removable storage disks, CD-ROMs, or any other form of storage medium known in the art.

[0105] Those skilled in the art will understand that the exemplary components, systems, and methods described in conjunction with the embodiments disclosed herein can be implemented in hardware, software, or a combination of both. Whether implemented in hardware or software depends on the specific application and design constraints of the technical solution. Those skilled in the art can use different methods to implement the described functions for each specific application, but such implementation should not be considered beyond the scope of this invention. When implemented in hardware, it can be, for example, electronic circuits, application-specific integrated circuits (ASICs), appropriate firmware, plug-ins, function cards, etc. When implemented in software, the elements of this invention are programs or code segments used to perform the desired tasks. The programs or code segments can be stored in a machine-readable medium or transmitted over a transmission medium or communication link via data signals carried in a carrier wave.

[0106] It should be clarified that the present invention is not limited to the specific configurations and processes described above and shown in the figures. For the sake of brevity, detailed descriptions of known methods are omitted here. In the above embodiments, several specific steps are described and shown as examples. However, the method process of the present invention is not limited to the specific steps described and shown. Those skilled in the art can make various changes, modifications, and additions, or change the order of steps, after understanding the spirit of the present invention.

[0107] In this invention, features described and / or illustrated for one embodiment may be used in the same or similar manner in one or more other embodiments, and / or combined with or in place of features of other embodiments.

[0108] The above description is merely a preferred embodiment of the present invention and is not intended to limit the present invention. For those skilled in the art, various modifications and variations of the embodiments of the present invention are possible. Any modifications, equivalent substitutions, improvements, etc., made within the spirit and principles of the present invention should be included within the protection scope of the present invention.

Claims

1. A method for detecting stance bias of a large language model, characterized in that, The method includes the following steps: The target language model generates unbiased news headlines about multiple dimensions of topics under multiple themes according to the unbiased generation requirements set by the input template of the first type of prompt words. According to the input template of the second type of prompt words corresponding to the multiple biases of each dimension of topic, multiple biased news headlines are generated for each bias. After vectorizing the unbiased news headlines and the biased news headlines using a pre-trained sentence embedding model, unbiased news headline vectors and biased news headline vectors are obtained. Then, based on cosine similarity, the similarity score between the unbiased news headline vectors corresponding to each topic dimension and multiple biased news headline vectors is calculated. The stance inclination and degree of stance inclination of the target large language model to be detected are determined based on the similarity scores between the unbiased news headline vector and the biased news headline vectors corresponding to each stance bias.

2. The method of claim 1, wherein the method further comprises: The method further includes: Obtain a benchmark dataset containing multiple multiple-choice questions, with each question option having a score; the multiple-choice questions pertain to multiple dimensions of topics across multiple subject categories. The single-choice question is input into the target language model to be detected and the options are output. The basic stance tendency of the target language model is obtained by comparing the score of the output option with the preset judgment criteria. The basic stance tendency is compared with the stance tendency determined based on the similarity score. If they match, they are both used as the stance tendency output of the target large language model. If they do not match, a new single-choice question is obtained from the Benchmark dataset and input into the target large language model to obtain the option score. This score is then compared with the preset judgment criteria to obtain the basic stance tendency.

3. The method of claim 1, wherein the determining of the stance of the large language model is based on a plurality of keywords. The steps of the target language model to generate news headlines about multiple dimensions of topics under multiple categories based on the prompt word input template include: Identify multiple topics across various themes that require news headlines; Determine the language style and headline format to be generated based on the target audience and applicable scenario; A general prompt word input template is developed to guide the large language model of the target to be detected to generate text content. The text expression form of the general prompt word input template is refined according to the target generation requirements of multiple dimensions of topics under multiple categories to obtain the prompt word input template. The prompt word input template is input into the target large language model to be detected and news titles of multiple dimensions under various topics are obtained; The news headline is subjected to bias detection according to the target generation requirements using a preset natural language processing tool, and a news headline that meets the target generation requirements is obtained.

4. The method of claim 1, wherein the determining of the stance of the large language model is based on a plurality of keywords. After vectorizing the unbiased news headlines and the biased news headlines using the pre-trained sentence embedding model, the resulting unbiased news headline vectors and biased news headline vectors include: The input unbiased news headlines and biased news headlines are segmented into unbiased sub-words and biased sub-words; The biased subwords and the biased subwords are mapped to encoding sequences adapted to the target large language model to be detected, and the encoding sequences are input into the sentence embedding model to output unbiased subword vectors and biased subword vectors; The unbiased news headline vector and the biased news headline vector are obtained by performing pooling operations on the unbiased sub-word vector and the biased sub-word vector.

5. The method of claim 1, wherein the determining of the stance of the large language model is based on a plurality of keywords. The similarity score between the unbiased news headline vector and the biased news headline vector is calculated based on cosine similarity, expressed as follows: Where A represents the unbiased news headline vector, and B represents the biased news headline vector.

6. The method of claim 1, wherein the determining of the stance of the large language model is based on a plurality of keywords. The stance inclination and degree of stance inclination of the target large language model are determined based on the similarity scores between the unbiased news headline vector and the biased news headline vectors corresponding to each stance bias, including: The stance tendency of the target large language model to be detected is classified into the category to which the maximum similarity score belongs among multiple similarity scores, based on the maximum similarity method; According to the similarity score S of the unbiased news headline vector with the biased news headline vectors of two different stands S and S O The difference value is calculated, expressed as: D = S S - S O ; When the difference between the absolute value of the smallest difference value and 0 exceeds a set threshold, it is determined that the stance inclination of the target large language model to be detected is obvious.

7. The method of claim 1, wherein the determining of the stance of the large language model is based on a plurality of keywords. The method further includes: A stance bias report is compiled based on the similarity score, the stance bias, and the degree of the stance bias, and the stance bias of the target large language model to be detected is analyzed. The stance bias report includes: the statistical results of the similarity score, the stance bias, and the degree of the stance bias; the performance of the target large language model to be detected on different news headlines; the analysis of the reasons for the stance bias; and the improvement measures for the prompt word input template. 8.A system for detecting stance bias of a large language model, comprising a processor, a memory, and computer programs / instructions stored on the memory, wherein, The processor is configured to execute the computer program / instructions, and when the computer program / instructions are executed, the system implements the steps of the method as described in any one of claims 1 to 7.

9. A computer readable storage medium having stored thereon computer programs / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

10. A computer program product, comprising a computer program / instructions, characterized in that, When the computer program / instructions are executed by the processor, they implement the steps of the method as described in any one of claims 1 to 7.

Images