Method and system for automatically evaluating online debate

Abstract

The invention provides a method and a system for automatically evaluating online debate. The method comprises the following steps: defining a plurality of role knowledge dimensions; customizing a prompt template according to the plurality of defined role knowledge dimensions to generate a role knowledge guide prompt, wherein the role knowledge guide prompt comprises an instruction, a situation demonstration and a situation and a discussion; inputting the role knowledge guiding prompt into a pre-trained language model to guide role knowledge; connecting the guided role knowledge with the situation and the argument in the prompt to construct a role knowledge alignment prompt; and utilizing the role knowledge alignment prompt to finely adjust a debate evaluation model to evaluate the persuasion / influence of the online debate argument.

Description

Technical Field

[0001] The present invention generally relates to techniques for fine-tuning language models using prompts. More specifically, the present invention relates to a method and system for automatically evaluating online debates using role knowledge alignment prompts. Background Art

[0002] Debate is the process of exchanging views or persuading others about a particular issue. Specifically, debate has been a cornerstone of human communication and decision-making throughout history, providing a platform for individuals and groups to express their ideas, argue their positions, and persuade others. The art of debate is crucial in a variety of fields such as politics, education, law, and everyday decision-making. The persuasiveness of arguments depends not only on the use of language, but also on the characteristics of the communicator, such as tone of voice, body language, and perceived credibility, highlighting the complexity of effective communication and the multifaceted nature of the persuasion process.

[0003] In the field of natural language processing (NLP) and computational argumentation, there is growing research interest in developing computational methods that can automatically assess qualitative features of arguments. The impact and persuasiveness of arguments are crucial qualitative features, and a large amount of research has been conducted to develop computational methods for identifying the impact and persuasiveness of natural language arguments in public forums. ChatGPT's impressive simulations of the open world show the potential of large pre-trained language models to act as individuals to provide roles and exert unique influences based on different background knowledge, enhancing the quality and effectiveness of debates.

[0004] Some computational argumentation research uses corpora collected from online argumentation sources such as Reddit's Change My View (CMV) subforum to assess the qualitative influence of arguments. However, estimating the influence or persuasiveness of arguments covering a variety of topics requires broader knowledge than just understanding the semantic meaning of language-based arguments in conversations. Many literatures document the importance and effects of various aspects such as surface text, social communication, argument-related features, characteristics of sources and audiences, sequential ordering of arguments, and stylistic features in determining persuasiveness. In addition to the aforementioned features, pragmatics and discourse context have also been applied to argument analysis. They have experimentally demonstrated that historical arguments benefit model performance to a certain extent. In addition, there are also some studies on how the context and dynamic development of argumentative conversations affect the relative persuasiveness during debates.

[0005] Traditional argument assessment tasks mainly study various aspects of knowledge. The influence and persuasiveness of arguments are closely related not only to language properties, but also to the characteristics of the communicator, including the source (speaker), prior beliefs, argument structure, and the influence of discourse context.

[0006] Social psychology research has identified audience as one of the factors that influence the persuasiveness of arguments. Integrating psychological principles into the study of debate can provide a deeper understanding of how to structure and present arguments more effectively to achieve desired persuasive outcomes.

[0007] Although researchers have paid much attention to the application of various aspects of language (such as linguistic features and discourse structure) to online debates, they have not yet been explored in combination with psychology due to their difficulty and complexity. Computational research on personas mainly comes from the field of dialogue, emphasizing the importance of personality information in dialogue. However, computationally extracting persona knowledge from analytical psychology to evaluate the influence / persuasiveness of online arguments has not been fully explored, except for annotating human characters through analytical psychology. Summary of the invention

[0008] We observed excellent simulations of LLMs in the open world, indicating that large pre-trained language models are able to play roles and exert unique influences based on different background knowledge. The combination of psychological principles with language features and discourse structure makes up for the shortcomings in current debate research and practice. Therefore, we proposed a role knowledge alignment framework for evaluating argument quality from a psychological perspective, explored the role of role knowledge in online debates for the first time, and injected psychological information into smaller language models through prompt tuning to classify the influence and persuasiveness of arguments and better perform online debate evaluation tasks.

[0009] According to one aspect of the present invention, a method for automatically evaluating online debates is provided. The method comprises: defining multiple role knowledge dimensions; customizing a prompt template according to the multiple defined role knowledge dimensions to generate a role knowledge guidance prompt, wherein the role knowledge guidance prompt comprises instructions, situation demonstration, and situation and argument; inputting the role knowledge guidance prompt into a pre-trained language model to induce role knowledge; connecting the induced role knowledge and the situation and argument in the prompt to construct a role knowledge alignment prompt; and using the role knowledge alignment prompt to fine-tune a debate evaluation model to evaluate the persuasiveness / influence of online debate arguments.

[0010] In some embodiments, the use of the role knowledge alignment prompt to fine-tune the debate evaluation model to evaluate the persuasiveness / influence of online debate arguments includes: formulating a classification problem of the persuasiveness / influence of online debate arguments; using the role knowledge alignment prompt to formulate the classification problem of the persuasiveness / influence of online debate arguments as a conditional generation task to generate learnable continuous tokens; updating the learnable continuous tokens through back propagation to generate output tokens; and mapping the generated output tokens to class labels of the persuasiveness / influence of online debate arguments.

[0011] According to another aspect of the present invention, a system for automatically evaluating online debates is provided. The system includes a memory and a processor, characterized in that: the memory is configured to store multiple predefined role knowledge dimensions, a pre-trained language model, and a debate evaluation model; the processor is configured to: customize a prompt template according to the multiple defined role knowledge dimensions to generate a role knowledge guidance prompt, wherein the role knowledge guidance prompt includes instructions, situation demonstrations, and situations and arguments; input the role knowledge guidance prompt into a pre-trained language model to guide role knowledge; connect the guided role knowledge and the situation and arguments in the prompt to construct a role knowledge alignment prompt; and use the role knowledge alignment prompt to fine-tune the debate evaluation model to evaluate the persuasiveness / influence of online debate arguments.

[0012] In some embodiments, the processor is further configured to: formulate a classification problem of the persuasiveness / influence of online debate arguments; formulate the classification problem of the persuasiveness / influence of online debate arguments as a conditional generation task using the role knowledge alignment prompt to generate a learnable continuous token; update the learnable continuous token by backpropagation to generate an output token; and map the generated output token to a class label of the persuasiveness / influence of the online debate arguments. BRIEF DESCRIPTION OF THE DRAWINGS

[0013] Embodiments of the present invention are described in more detail below with reference to the accompanying drawings, in which:

[0014] Figure 1 shows role knowledge derived from an LLM using a customized prompt template according to an embodiment of the present invention;

[0015] Figure 2 A role knowledge alignment prompting framework for automatically evaluating online debates according to an embodiment of the present invention is shown;

[0016] Figure 3 The evaluation of five aspects of role knowledge according to an embodiment of the present invention is shown;

[0017] Figure 4 is a customized prompt template used to derive role knowledge from LLM in an embodiment of the present invention;

[0018] Figure 5 shows a plurality of PersonalPrompt templates during a template search in an embodiment of the present invention; and

[0019] Figure 6 It shows the role knowledge derived from another LLM using a customized prompt template according to one embodiment of the present invention.

[0020] Fig. 7A and 7B A flow chart of a method for automatically evaluating an online debate according to an embodiment of the present invention is shown. DETAILED DESCRIPTION

[0021] In the following description, methods and systems for automatically evaluating online debates are described as preferred examples. It should be understood by those skilled in the art that modifications, including additions and / or substitutions, may be made without departing from the scope and spirit of the invention. Certain details may be omitted so as not to obscure the invention; however, this disclosure is written to enable those skilled in the art to practice the teachings herein without undue experimentation.

[0022] According to another aspect of the present invention, a method S100 for automatically evaluating an online debate is provided. Fig. 7A , the method S100 comprises the steps of:

[0023] S102: Define multiple role knowledge dimensions;

[0024] S104: Customizing a prompt template according to the multiple defined role knowledge dimensions to generate a role knowledge guidance prompt, wherein the role knowledge guidance prompt includes instructions, situation demonstration, and situation and argument;

[0025] S106: inputting the role knowledge guidance prompt into a pre-trained language model to guide role knowledge;

[0026] S108: Connecting the derived role knowledge with the context and argument in the prompt to construct a role knowledge alignment prompt; and

[0027] S110: Fine-tune the debate evaluation model using the role knowledge alignment prompt to evaluate the persuasiveness / influence of online debate arguments.

[0028] refer to Figure 7B In some embodiments, step S110 includes:

[0029] S1102: Formulate a taxonomy of persuasiveness / influence of online debate arguments;

[0030] S1104: using the knowledge alignment prompt to formulate the classification problem of the persuasiveness / influence of the online debate arguments as a conditional generation task to generate learnable continuous tokens;

[0031] S1106: updating the learnable continuous tokens by back propagation to generate output tokens; and

[0032] S1108: Mapping the generated output tokens into class labels of the persuasiveness / influence of online debate arguments.

[0033] Figure 4 A customized prompt template for eliciting character knowledge from an LLM according to an embodiment of the present invention is shown. The prompt template shown includes three parts: instructions, situation demonstration, situation and argument. The instructions describe the purpose of the task and define the response we need from the language model. The situation demonstration demonstrates the situation, argument and knowledge paragraphs of multiple characters. The knowledge paragraph of each character contains the position, argument of the character on the situation and argument, as well as the character traits and intentions of the character. The situation and argument of the prompt template are different from the situation and argument of the situation demonstration. The LLM can be any language model that includes advanced NLP technology, for example, GPT 3.5, GPT 4 or GPT 4o.

[0034] According to another aspect of the present invention, a system for automatically evaluating online debates is also provided. The system includes: a digital platform that promotes virtual discussions and participant participation; a module that incorporates psychological principles into the structure and expression of arguments; and a component that uses a role knowledge alignment framework from a psychological perspective to perform argument quality assessment tasks. The prompt template can be customized in the module through the digital platform, and the component guides the role knowledge from the debate language model. The digital platform may include a keyboard, a mouse, a touchpad, a touch screen, or any other suitable input device. The module may include a memory and a processor (e.g., a central processing unit (CPU)). The component may include a graphics processing unit (GPU).

[0035] In some embodiments, the component may include a GPU dedicated to accelerating artificial intelligence (AI), high performance computing (HPC), data science, and graphics. For example, in some embodiments, the component may include at least two NVIDIA V100 Tensor Core GPUs with a 32GB configuration, and a large amount of computing resources can be trained in a few days and can be used for debate simulations.

[0036] In an embodiment of the present invention, role knowledge has multiple dimensions. In order to construct an efficient and task-specific role knowledge representation for the online debate argument task, we customize a specific prompt template to induce multi-dimensional role knowledge from a large language model. Figure 6 The role knowledge guided by the language model is presented in the Figure 4 Dimensions:

[0037] Character Stance: This dimension describes the character's stance (i.e., against, for, or neutral) regarding a given argument or situation. For example, Figure 1 The software engineer role is shown to have an opposing stance towards the given data.

[0038] Persona stances provide significant benefits by enhancing the effectiveness and persuasiveness of arguments in debates and discussions. Persona stances enable more tailored and impactful communication strategies by aligning arguments with the speaker’s psychological profile and characteristics. By taking into account factors such as the speaker’s tone, body language, credibility, and background knowledge, arguments are presented in a way that resonates more deeply with the audience. This personalized approach not only improves the clarity and expression of the message, but also increases the likelihood of persuading the audience. In addition, persona stances facilitate a better understanding of the audience’s perceptions and reactions, allowing speakers to dynamically adjust their arguments in real time, resulting in more engaging and productive interactions. Overall, integrating persona stances into the argumentation process significantly improves the quality and effectiveness of communication, making it a powerful tool in a variety of fields such as politics, education, and law.

[0039] Character Argument: This dimension presents the character's argument using his / her own background knowledge to support his / her position.

[0040] Persona arguments provide substantial benefits by tailoring arguments to fit the individual characteristics and psychological profiles of both the speaker and the audience. This personalized approach takes into account factors such as tone, body language, perceived credibility, and specific background knowledge, making arguments more credible and persuasive. By leveraging persona arguments, speakers can enhance their connection with their audience, increasing engagement and trust. This approach allows for dynamic adaptation of arguments in real time, ensuring that presentations remain impactful and relevant throughout the discussion. In addition, persona arguments facilitate a deeper understanding of audience responses, enabling more strategic and effective communication. This approach is particularly beneficial in areas such as marketing, education, and public speaking, where impact and persuasion are critical. Overall, persona arguments significantly improve the quality of discourse by making it more targeted, effective, and engaging.

[0041] Character: This dimension describes the inherent character traits that a character may display. For example, Figure 1 It shows that a computer science professor might have characteristics such as being knowledgeable and logical.

[0042] Personas provide significant benefits by allowing for the creation of customized and relevant communication strategies in a variety of contexts. These personas, derived from detailed psychological profiles and contextual information, enable speakers to craft arguments and messages that resonate deeply with their target audiences. By embodying specific traits such as tone of voice, body language, and credibility, personas help to build a strong connection with audiences, thereby enhancing engagement and trust. This personalized approach ensures that content is not only more persuasive, but also more memorable. In marketing, education, and customer service, personas can make interactions more dynamic and effective by addressing the unique needs and preferences of different audience segments. In addition, personas enable more precise and impactful storytelling, which is essential to attracting and retaining audience attention. Overall, personas significantly improve the effectiveness of communications by making them more authentic, engaging, and targeted.

[0043] Role Intent: This dimension outlines the external effects or outcomes that the role intends to achieve or accomplish in the upcoming time period. For example, a possible intent for a computer science professor is to "instruct and educate students on programming fundamentals and provide insight into the benefits of learning C++."

[0044] Persona intent provides significant benefits by ensuring that communications are purpose-driven and aligned with the goals of both the speaker and the audience. By clearly defining the intent behind each persona, speakers can craft messages that are more targeted, persuasive, and relevant to their target audience. This approach enhances the clarity and impact of communications because it takes into account the specific goals, motivations, and desired outcomes of both the speaker and the listener. Persona intent helps create arguments that are not only logically sound but also emotionally compelling, increasing the likelihood of achieving the desired effect, whether to inform, persuade, or engage. In marketing, education, and public relations, persona intent ensures that messages are strategically designed to meet the needs and expectations of the audience, resulting in more effective interactions and stronger connections. Overall, persona intent adds depth and direction to communication efforts, making them more impactful and goal-oriented.

[0045] Figure 1 and Figure 6 The role knowledge derived from the language model GPT-3.5 and the language model GPT-4 using the prompt template of the present invention are presented in Figure 1 , and each role knowledge includes the role stance, the role argument, the role character, and the role intention. It can be seen that the prompt template of the present invention is applicable to most LLMs.

[0046] In some embodiments of the present invention, in order to enhance the quality, effectiveness and usefulness of the generated character knowledge and solve the potential illusion problem in the generated content, the four dimensions corresponding to the knowledge generated by each character are scored according to the following criteria. Then, by inputting the character knowledge, LLM can provide an evaluation report containing the scores corresponding to the above dimensions.

[0047] Relevance: It determines whether the characters, arguments, and intentions are relevant to the argument and situation.

[0048] The item "relevance" in argumentative assessments provides significant benefits by ensuring that the content being assessed is directly related to the topic, situation, or user needs. By focusing on relevance, evaluators can determine how well the information or argument fits the topic and the interests of the audience, resulting in more meaningful and impactful communication. This criterion helps filter out irrelevant or off-topic content, ensuring that the discourse remains relevant and valuable. In educational settings, relevance ensures that instructional materials and discussions are closely aligned with learning objectives, thereby enhancing comprehension and retention. In marketing and customer service, relevance ensures that messages and solutions meet the specific needs and concerns of the target audience, thereby increasing satisfaction and engagement. Overall, incorporating relevance into argumentative assessments improves the effectiveness and efficiency of communication by maintaining a clear and purposeful focus, making interactions more productive and satisfying.

[0049] Fluency: It assesses the smoothness and comprehensibility of characters, arguments, and intentions.

[0050] The dimension “fluency” in debate assessment provides significant benefits by ensuring that content is easy to understand, well-structured, and grammatically correct. Assessing fluency helps identify and eliminate language errors, awkward wording, and unclear expressions that can hinder comprehension and reduce the overall impact of the message. In educational materials, marketing content, and customer communications, high fluency enhances readability and engagement, making information more accessible to a wider audience. By prioritizing fluency, evaluators can ensure that content flows well, facilitating better comprehension and retention of the information presented.

[0051] Coherence: It assesses whether the intention and arguments are consistent with the position.

[0052] The dimension “consistency” in the debate assessment ensures that the content maintains a consistent tone, style, and theme throughout. Consistency is critical to building trust and credibility because it prevents confusion and reinforces the intended theme. In branding, educational materials, and professional documents, consistency helps create a coherent narrative, making the information more reliable and professional. By assessing consistency, content creators can avoid inconsistencies and discrepancies, ensuring that audiences receive a clear and coherent message that aligns with their expectations and the overall purpose of the communication.

[0053] Credibility: It measures the plausibility and credibility of intentions and arguments;

[0054] The item "Credibility" in the debate assessment provides the benefit of ensuring that the content is credible and logically sound. Assessing credibility helps verify that the arguments, claims, and information presented are credible and based on evidence or logical reasoning. This is particularly important in academic writing, scientific research, and persuasive communication where the validity of the content is critical. By focusing on credibility, evaluators can enhance the reliability and persuasiveness of the content, thereby fostering trust and acceptance among the audience.

[0055] Usefulness: This measures whether the data produced helps to establish the persuasiveness of the argument;

[0056] The item "usefulness" in debate evaluation ensures that the content provides real value and meets the needs of the audience. Assessing usefulness helps determine whether the information or arguments presented are relevant, actionable, and beneficial to the target audience. In educational settings, customer support, and informational content, usefulness ensures that the audience receives valuable insights and solutions that meet their specific concerns and interests. By prioritizing usefulness, evaluators can enhance the overall impact and effectiveness of the content, making it more likely to achieve its intended purpose.

[0057] Harmfulness: It estimates whether the knowledge generated contains harmful and toxic language or words.

[0058] The item "Harmfulness" in the debate assessment helps identify and mitigate any potential negative impacts of content. Assessing harmfulness involves examining content that may be offensive, misleading, or damaging to individuals or groups. This is essential to maintaining ethical standards and social responsibility in communication. By focusing on harmfulness, assessors can ensure that content is respectful, accurate, and safe to consume, thereby protecting the health of audiences and preventing negative impacts. This assessment promotes a more proactive and responsible approach to content creation and distribution.

[0059] For example, Figure 3 An evaluation of character knowledge generated by ChatGPT is presented. The harmful aspects of these character knowledge are zero. This may be attributed to fine-tuning ChatGPT through reinforcement learning with human feedback method, preventing ChatGPT from generating harmful language without intentional attacks.

[0060] Some embodiments of the present invention have demonstrated that the pre-trained language model (PLM) of this embodiment can effectively understand label semantics and task objectives through prompt learning. We use indicators to separate "context" and "argument" and indicate the model's prediction of the winner of the debate or the potential influence level of the argument.

[0061] Additionally, in these embodiments, we incorporate the role knowledge generated by ChatGPT as “background” prior to the situation, which provides a broader and more comprehensive perspective on the debate process.

[0062] To ensure the generation of high-quality character knowledge, we not only provide data (i.e., context and arguments) and carefully crafted instructions, but also context demonstrations that leverage character knowledge to guide prompts (I) to cue ChatGPT to generate 4D knowledge in a specific format.

[0063] In these examples, we adopt a more efficient approach called soft-hint tuning instead of fine-tuning the entire PLM architecture. We introduce learnable "continuous tokens" at the beginning of the input, allowing them to be updated via back-propagation. We formulate the classification problem as a conditional generation task using the role-knowledge alignment hint (II) (i.e., role-knowledge guided hint), where the generated tokens are then mapped to class labels.

[0064] For example, PersonaPrompt is a prompt tuning for Flan-T5, which uses instructions to tune the T5 model, and we also validate our approach on two debate evaluation models (T5-base and T5-large) by setting the learnable prompt length to 20.

[0065] For the argument influence classification task, training was implemented with 30,000 training steps using cross entropy loss, which selects the debate evaluation model that produces the best performance on the validation set. We used the Adafactor optimizer with different learning rate ranges for different dataset settings. The batch size and maximum input sequence were 4 and 512, respectively. The maximum generated sequence length of the encoder was 10. Our debate evaluation model was run on two 32GB NVIDIAV100 GPUs. The running time for T5-base was about 8 hours, while the running time for T5-large was about 22 hours. The frozen pre-trained T5 model was downloaded from HuggingFace, and our model inheritance and modification was downloaded from OpenPrompt.

[0066] In other words, previous studies on language modeling have demonstrated that PLM can effectively understand label semantics and task objectives through prompt learning. Figure 2 As shown, the indicators separate “situation” and “argument” and indicate the model’s prediction of the winner of the debate or the potential level of influence of the argument.

[0067] In addition, the role knowledge generated by ChatGPT is the “background” before the situation, which provides a broader and more comprehensive perspective on the debate process.

[0068] To ensure high-quality characters, we provide not only data (i.e., scenarios and arguments) and carefully crafted instructions, but also scenario demonstrations (ref. Figure 4 ), using role knowledge to guide prompts (I) to prompt ChatGPT to generate four-dimensional knowledge in a specific format.

[0069] The PLM is processed by soft-hint tuning. In these embodiments, "continuous tokens" are included at the beginning of the input, allowing them to be updated via back-propagation. We formulate the classification problem as a conditional generation task using knowledge alignment hints (II), where the generated tokens are subsequently mapped to class labels.

[0070] PersonaPrompt is a prompt tuning for Flan-T5, which uses instructions to tune the T5 model, and we also validate our method on two model scales, including T5-base and T5-large. We set the learnable prompt length to 20. For the argument influence classification task, training is implemented with 30,000 training steps using cross entropy loss, which selects the model that produces the best performance on the validation set. We adopted the Adafactor optimizer with different learning rate ranges for different dataset settings. The batch size and maximum input sequence are 4 and 512 respectively. The maximum generated sequence length of the encoder is 10. Our model is run on two 32GB NVIDIA V100 GPUs. The running time of T5-base is about 8 hours, while the running time of T5-large is about 22 hours. The customized prompt template is Figure 5 Download the frozen pre-trained T5 model from HuggingFace, and download the inheritance and modification of our model from OpenPrompt.

[0071] The present invention also relates to a comprehensive method for enhancing the debate process by integrating advanced technology and psychological principles, which injects psychological information into smaller language models through prompt tuning, so that the smaller language models can more effectively evaluate the persuasiveness of arguments.

[0072] In some embodiments of the present invention, a framework is introduced that combines role knowledge with argument quality assessment. This framework evaluates arguments based on psychological principles, taking into account factors such as the speaker's tone, body language, and perceived credibility that are critical to determining the persuasiveness of an argument. The present invention further fills a gap in current debate research and practice by combining language features and discourse structure with psychological insights, providing deeper insights into effective argument evaluation.

[0073] The present invention also takes pragmatics and discourse context into account when analyzing arguments, fully ensuring that all relevant factors are considered when evaluating the quality of an argument.

[0074] The present invention further explores how the context and dynamic progression of argumentative dialogues affect relative persuasiveness. Studies have shown that the evolving nature of dialogues and contextual factors significantly influence the effectiveness of arguments.

[0075] The present invention can be applied to digital platforms equipped with tools to facilitate virtual discussions, incorporating features such as real-time feedback, interactive interfaces, and enhanced accessibility options to improve the overall debate experience, expand the range of platform participants, and thus make debates easier to understand and more attractive.

[0076] To verify the effectiveness of role knowledge in debate tasks, in this embodiment, we conducted experiments on two tasks: argument influence classification in the Kialo dataset and argument persuasiveness prediction in the DDO dataset.

[0077] Table 1: Mean and standard deviation of the performance of different models for argument influence classification. KNOWLEDGE indicates the combination with the generated role knowledge.

[0078] Table 2: Model performance on the argument persuasiveness prediction task, where the upper part of the baseline model is a logistic regression (LR) model with language features, user information and argument structure.

[0079] In these tables, precision is a metric used to measure the accuracy of the persuasiveness predictions made by the model. It is calculated as the number of true positive results divided by the sum of true positive results and false positive results. Simply put, precision tells us how many of the predicted positive results are actually correct. A high precision indicates that the model has a low false positive rate, meaning that it is good at avoiding incorrect persuasiveness predictions.

[0080] Recall, also known as sensitivity or true positive rate, measures the ability of a model to identify all positive examples within a data set. It is calculated as the number of true positive results divided by the sum of true positive results and false negative results. Recall tells us how many actual positive examples the model correctly identified. A high recall indicates that the model detects positive cases well, even if the model sometimes contains false positives.

[0081] The F1 score is a metric that combines both precision and recall into a single metric. It is the harmonic mean of precision and recall, providing a balance between the two. The F1 score is particularly useful when we need to find a balance between precision and recall, especially when we want to ensure that both false positives and false negatives are not too high. A high F1 score indicates that the model has a good balance between precision and recall.

[0082] Tables 1 and 2 summarize the main results of two online debate tasks, including the argument influence classification task and the argument persuasiveness task, from which we draw the following conclusions.

[0083] First, our method (i.e., PersonaPrompt) significantly outperforms all baseline methods in both tasks and achieves state-of-the-art (SOTA) performance in the argument influence classification task. Specifically, our method (based on T5-base) has an F1 score of at least 4.45% in the argument influence classification task, outperforming the previous SOTA DisCOC.

[0084] Second, our model achieves a significant improvement of 4.36% F1 score and 3.30% accuracy compared to fine-tuning the T5-base (no role knowledge) model in the Kialo dataset and the DDO dataset. This proves that our approach effectively exploits role knowledge, perceives this specific knowledge through its relevance to data arguments and context, and ultimately enhances the ability of the T5 model to undertake this challenging task.

[0085] Third, all model fine-tuning or prompt tuning using knowledge improves over the original tuning to some extent. In particular, PersonPrompt (T5-base) achieves a 3.51% F1 score gain in performance over the original tuning without role knowledge. It shows the effectiveness of the generated role knowledge for the online debate quality assessment task.

[0086] The functional units and modules of the system according to the embodiments disclosed herein can be implemented using a computing device, a computer processor or an electronic circuit system, including but not limited to an application specific integrated circuit (ASIC), a field programmable gate array (FPGA), a microcontroller and other programmable logic devices configured or programmed according to the teachings of the present disclosure. The computer instructions or software codes running in the computing device, computer processor or programmable logic device can be easily made by practitioners in the field of software or electronic technology based on the teachings of the present disclosure.

[0087] All or part of the method according to the embodiment may be executed in one or more computing devices including a server computer, a personal computer, a laptop computer, a mobile computing device (eg, a smart phone), and a tablet computer.

[0088] Embodiments may include computer storage media, transitory and non-transitory memory devices having stored therein computer instructions or software codes that may be used to program or configure a computing device, a computer processor, or an electronic circuit system to perform any of the processes of the present invention. Storage media, transitory and non-transitory memory devices may include, but are not limited to, floppy disks, optical disks, Blu-ray disks, DVDs, CD-ROMs and magneto-optical disks, ROMs, RAMs, flash memory devices, or any type of media or device suitable for storing instructions, codes, and / or data.

[0089] Each of the functional units and modules according to various embodiments may also be implemented in a distributed computing environment and / or a cloud computing environment, where all or part of a machine instruction is executed in a distributed manner by one or more processing devices interconnected by a communication network, such as an intranet, a wide area network (WAN), a local area network (LAN), the Internet, and other forms of data transmission media.

[0090] The foregoing description of the present invention has been provided for the purpose of illustration and description. The present invention is not intended to be exhaustive or to limit the present invention to the precise form disclosed. Many modifications and variations will be apparent to those skilled in the art.

[0091] The embodiment was chosen and described in order to best explain the principles of the invention and its practical application, thereby enabling others skilled in the art to understand the invention for various embodiments and with various modifications as are suited to the particular use contemplated.

Claims

1. A method for automatically evaluating online debates, characterized in that include: Define multiple role knowledge dimensions; customizing prompt templates according to the plurality of defined role knowledge dimensions to generate role knowledge guiding prompts, wherein the role knowledge guiding prompts include instructions, situation demonstrations, and situations and arguments; Inputting the role knowledge guidance prompt into a pre-trained language model to induce role knowledge; connecting the elicited role knowledge with the context and arguments in the prompt to construct a role knowledge alignment prompt; as well as The role knowledge alignment prompts are utilized to fine-tune the debate evaluation model to evaluate the persuasiveness / influence of online debate arguments.

2. The method according to claim 1, characterized in that The method of fine-tuning the debate evaluation model using the role knowledge alignment prompt to evaluate the persuasiveness / influence of online debate arguments includes: Formulate a taxonomy of persuasiveness / influence of online debate arguments; formulating the classification problem of the persuasiveness / influence of the online debate arguments as a conditional generation task using the knowledge alignment prompt to generate learnable continuous tokens; updating the learnable continuous tokens by back-propagation to generate output tokens; and The generated output tokens are mapped to class labels of the persuasiveness / influence of online debate arguments.

3. The method according to claim 2, characterized in that The plurality of role knowledge dimensions include a role stance, wherein the role stance describes the role's stance regarding the situation and the argument.

4. The method according to claim 3, characterized in that The plurality of role knowledge dimensions include role arguments, wherein the role arguments present arguments supporting the role's position.

5. The method according to claim 3, characterized in that: The plurality of role knowledge dimensions include role traits, wherein the role traits describe the inherent character traits exhibited by the role.

6. The method according to claim 3, characterized in that The plurality of role knowledge dimensions include role intent, wherein the role intent summarizes the external effects or results that the role is intended to achieve or accomplish.

7. The method according to claim 1, characterized in that Further included is scoring the relevance of the knowledge about the role.

8. The method according to claim 1, characterized in that Further included is scoring fluency regarding knowledge of said role.

9. The method according to claim 1, characterized in that: Further included is scoring consistency of knowledge about the role.

10. The method according to claim 1, characterized in that Further included is scoring the credibility of the knowledge about the character.

11. The method according to claim 1, characterized in that: Further included is scoring the usefulness of the knowledge about the role.

12. The method according to claim 1, characterized in that Further included is scoring the harmfulness of knowledge about the character.

13. A system for automatically evaluating online debates, comprising a memory and a processor, characterized in that: The memory is configured to store a plurality of predefined role knowledge dimensions, a pretrained language model, and a debate evaluation model; The processor is configured as follows: customizing prompt templates according to the plurality of defined role knowledge dimensions to generate role knowledge guiding prompts, wherein the role knowledge guiding prompts include instructions, situation demonstrations, and situations and arguments; Inputting the role knowledge guidance prompt into a pre-trained language model to induce role knowledge; connecting the elicited role knowledge with the context and arguments in the prompt to construct a role knowledge alignment prompt; as well as The role knowledge alignment prompts are used to fine-tune the debate evaluation model to evaluate the persuasiveness / influence of online debate arguments.

14. The system according to claim 13, characterized in that The processor is further configured to: Formulate a taxonomy of persuasiveness / influence of online debate arguments; The persuasiveness / influence classification problem of the online debate arguments is formulated as a conditional generation task using the role knowledge alignment prompt to generate learnable continuous tokens; updating the learnable continuous tokens by back-propagation to generate output tokens; as well as The generated output tokens are mapped to class labels of the persuasiveness / influence of online debate arguments.

15. The system according to claim 14, characterized in that The plurality of role knowledge dimensions include a role stance, wherein the role stance describes the role's stance regarding the situation and the argument.

16. The method according to claim 15, characterized in that The plurality of role knowledge dimensions include role arguments, wherein the role arguments present arguments supporting the role's position.

17. The system according to claim 15, characterized in that The plurality of role knowledge dimensions include role traits, wherein the role traits describe the inherent character traits exhibited by the role.

18. The system according to claim 15, characterized in that The plurality of role knowledge dimensions include role intent, wherein the role intent summarizes the external effects or results that the role is intended to achieve or accomplish.

19. The system according to claim 13, characterized in that The processor is further configured to score the relevance of the knowledge about the role.

20. The system according to claim 13, characterized in that The processor is further configured to score fluency regarding the character's knowledge.

21. The system according to claim 13, characterized in that The processor is further configured to score consistency regarding the role knowledge.

22. The system according to claim 13, characterized in that The processor is further configured to score the credibility of the knowledge about the role.

23. The system according to claim 13, characterized in that The processor is further configured to score the usefulness of the knowledge about the role.

24. The system according to claim 13, characterized in that The processor is further configured to score the harmfulness of the knowledge about the role.

Images