A method for evaluating learning effectiveness of a learner in human-computer collaborative writing
Patent Information
- Application Number
- CN202611019982.8
- Authority / Receiving Office
- CN · China
- Patent Type
- Applications(China)
- Current Assignee / Owner
- Filing Date
- 2026-07-09
- Publication Date
- 2026-09-25
AI Technical Summary
这给学习成效评估带来了根本性挑战：教师面对学生提交的成果，难以判断该成果在多大程度上反映了学生自身的学习与掌握，还是仅仅是对人工智能输出的复制粘贴
（1）突破了现有AI内容检测工具仅能判断文本是否由AI生成的局限。本发明不仅识别文本的AI来源，更重要的是通过分析学习者对AI内容的加工行为，判断学习者是否在此过程中真正理解和掌握了知识，从而回答“学生是否真正学到了”这一核心问题。
Smart Images
Figure CN122819192A_ABST
Abstract
Description
Technical Field
[0001] This invention relates to educational technology and natural language processing, and more particularly to a method for evaluating learner learning outcomes in human-computer collaborative writing. Background Technology
[0002] With the popularization of generative artificial intelligence technology, students rely heavily on AI to generate text when completing writing assignments. This poses a fundamental challenge to the assessment of learning outcomes: teachers find it difficult to determine the extent to which student submissions reflect their own learning and mastery, or are merely copies and pastes of AI output.
[0003] Existing AI content detection tools can only determine whether text was generated by artificial intelligence and provide a "probability of AI generation," but they cannot answer the more crucial question: even if the content uses artificial intelligence, have students truly understood and mastered the knowledge in the process? Detecting that "it was written by artificial intelligence" does not mean "students haven't learned anything"; and vice versa.
[0004] Existing plagiarism detection tools rely on literal text similarity and cannot identify the deep cognitive processing learners perform on AI content. Learners may reconstruct the AI's suggestions in their own words after understanding them, and this "learning" will be ignored; or they may only make superficial rewritings by replacing synonyms, and this "not learning" may escape detection.
[0005] Existing evaluation methods only focus on the final text output, ignoring the process data of learners interacting with artificial intelligence (such as dialogue records and modification trajectories), which contain key information for judging whether learning has truly occurred.
[0006] Therefore, there is an urgent need for a method that can accurately assess learners' actual learning outcomes with the assistance of artificial intelligence. Summary of the Invention
[0007] Purpose of the invention: The purpose of this invention is to provide a method for evaluating learners' learning effectiveness in human-computer collaborative writing. By collecting and analyzing process data of learners' interaction with artificial intelligence, the method quantifies and distinguishes between deep processing (learners autonomously reconstructing content after understanding it, corresponding to real learning) and superficial copying (learners directly copying or only making superficial modifications, corresponding to no learning), thereby evaluating the learners' true learning effectiveness.
[0008] Technical Solution: A method for evaluating learner learning outcomes in human-computer collaborative writing, comprising the following steps: Step 1: Collect data on the human-computer interaction process; During the learner's completion of the AI-assisted writing task, three types of data are automatically collected: AI dialogue data, including the instructions issued by the learner to the AI system and all generated content returned by the AI system; text operation data, including the operation type and operation timestamp when the learner imports the AI-generated content into their own text, the operation type including direct pasting, manual re-entry, and partial editing; and final text data, that is, the final result text submitted by the learner. Step Two: Establish a source mapping between AI content and the final text; compare each text segment in the final text with the AI-generated content to trace its source, classifying it into one of the following categories: direct AI source, i.e., the literal similarity with a certain AI-generated content is higher than a set first threshold; AI rewritten source, i.e., the semantic similarity with a certain AI-generated content is higher than a set second threshold, but the literal similarity is lower than the first threshold, showing semantic similarity but large literal differences; learner original, i.e., there is no obvious correspondence with any AI-generated content; among which, semantic similarity is calculated using semantic representation techniques such as sentence vectors, and literal similarity is calculated using string matching techniques such as edit distance; Step 3: Calculate the cognitive processing depth index; Based on the source tracing results of Step 2 and the operational data of Step 1, calculate the cognitive processing depth score for each text fragment originating from artificial intelligence: For fragments directly from artificial intelligence, the cognitive processing depth is lowest when the operation method is direct pasting, and it is judged as superficial copying; the cognitive processing depth is still low when the operation method is manual re-inputting, and it is still judged as superficial copying; For fragments rewritten by artificial intelligence, by detecting the degree of change of the rewritten fragment relative to the original text of artificial intelligence in syntactic structure, information organization order and argumentation logic, distinguish between superficial synonym replacement that only replaces words and structural reconstruction that has undergone substantial changes. The former has a medium cognitive processing depth and is judged as partial processing, while the latter has a high cognitive processing depth and is judged as deep processing; For fragments created by learners, the cognitive processing depth is the highest and is judged as deep processing; Step 4: Generate learning effectiveness assessment results; comprehensively analyze the processing depth of all text fragments, calculate the proportion of deep processing, the proportion of superficial copying, and the overall learning effectiveness score, and generate a visual diagnostic report, highlighting the true contribution distribution of learners by marking different source fragments in the final text with different colors.
[0009] Furthermore, in step one, text operation data is collected by listening to operating system clipboard events and keyboard input events in the learner's editing interface. For each text import operation, the operation type identifier, operation timestamp, imported text content, and insertion position in the learner's text are recorded.
[0010] Furthermore, in step two, the semantic similarity is calculated as follows: a pre-trained sentence vector model is used to map text segments into sentence vector representations, and the semantic similarity is obtained by calculating the cosine similarity between two sentence vectors; the literal similarity is calculated as follows: the Levenshtein edit distance algorithm is used to divide the edit distance between two strings by the length of the longer string to obtain the normalized literal similarity.
[0011] Furthermore, in step two, the source mapping identifies text fragments from which artificial intelligence rewrites them by combining semantically similar but literally different features. That is, fragments with semantic similarity higher than the second threshold and literal similarity lower than the first threshold are classified as artificial intelligence rewrite sources, in order to distinguish them from direct artificial intelligence sources with literal similarity higher than the first threshold.
[0012] Furthermore, in step three, the method for determining structural reconstruction is as follows: perform syntactic dependency analysis on the rewritten fragment and the original AI text respectively, extract their respective syntactic structures, and detect the degree of change of the rewritten fragment relative to the original AI text in terms of syntactic structure, information organization order, and argumentation logic. If substantial changes occur, it is determined to be structural reconstruction; if only word substitution occurs while the syntactic structure and information organization order remain basically unchanged, it is determined to be superficial synonym substitution.
[0013] Furthermore, in step four, the deep processing ratio is the proportion of the sum of the word counts of structurally reconstructed segments and learner-original segments to the total word count of the entire text; the surface copying ratio is the proportion of the sum of the word counts of directly pasted segments and manually re-entered segments to the total word count of the entire text.
[0014] Furthermore, in step four, the method for calculating the comprehensive score of learning effectiveness is as follows: each text segment is assigned a score according to the depth of cognitive processing, with direct pasting assigned the lowest score, manual re-entry assigned a lower score, surface synonym replacement assigned a medium score, structural reconstruction assigned a higher score, and learner originality assigned the highest score. The scores of each segment are weighted and summed according to the number of words, and then divided by the total number of words in the whole text to obtain a comprehensive score between 0 and 1.
[0015] The present invention also provides a computer-readable storage medium having a computer program stored thereon, wherein when the computer program is executed by a processor, the steps of the above-described method for evaluating learner learning outcomes in human-computer collaborative writing are implemented.
[0016] Beneficial effects: (1) This invention breaks through the limitation of existing AI content detection tools that can only determine whether the text was generated by AI. This invention not only identifies the AI source of the text, but more importantly, it analyzes the learner's processing behavior of AI content to determine whether the learner has truly understood and mastered the knowledge in the process, thereby answering the core question of "whether the student has truly learned".
[0017] (2) It breaks through the limitations of existing plagiarism detection tools that only detect plagiarism based on literal similarity. This invention can identify the deep cognitive processing of AI content by learners through the combination of semantic similarity and literal similarity, distinguish between superficial synonym replacement and structural reconstruction, avoid misjudging the situation of "learning and reconstructing in one's own words" as plagiarism, and also avoid missing the situation of "only doing synonym replacement".
[0018] (3) It realizes process evaluation. This invention collects complete process data of learners' interaction with AI, including dialogue records, text introduction operation types and timestamps, rather than just analyzing the final text results, so as to obtain key information to determine whether learning has really occurred.
[0019] (4) It provides quantitative learning effectiveness evaluation indicators. Through the hierarchical quantification of cognitive processing depth (progressive scoring from direct copying to structural reconstruction to originality), the proportion of deep processing, the proportion of superficial copying, and the comprehensive score of learning effectiveness are calculated, providing teachers with an objective and comparable evaluation basis.
[0020] (5) Generate a visual diagnostic report. Different source segments in the final text are marked with different colors for easy reference. Teachers and students can understand the true distribution of learners' contributions at a glance. This helps teachers to carry out targeted teaching interventions and also helps learners to reflect on and improve themselves. Attached Figure Description
[0021] Figure 1 This is a schematic diagram of the method flow of the present invention. Detailed Implementation
[0022] To make the technical solution of the present invention clearer, the present invention will be further described in detail below with reference to the accompanying drawings and specific embodiments.
[0023] Example 1
[0024] This example is applied to a university English writing course, requiring students to complete an argumentative essay on "The Social Impact of Artificial Intelligence," and allowing the use of AI-assisted writing tools. Figure 1 As shown, this method executes steps one through four in sequence.
[0025] Step 1: Collect data on the human-computer interaction process. Throughout the learner's use of the AI writing assistant, three types of data are automatically collected through the writing platform plugin: (1) Artificial intelligence dialogue data: The learner sends prompt words to the large language model each time and the model returns the complete generated text, which are organized into a dialogue sequence in chronological order. In this embodiment, a learner conducted a total of 8 rounds of dialogue, each round including prompt words and model responses.
[0026] (2) Text operation data: By listening to the operating system clipboard event and keyboard input event in the editing interface, the operation type (direct paste, manual re-entry, partial editing) and operation timestamp of each text import operation are recorded. In this embodiment, a total of 23 text import operations are recorded, including 10 direct pastes, 7 manual re-entrys, and 6 partial editings.
[0027] (3) Final text data: Learners submit the final text of their results, which is 1200 words long and contains 15 sentences.
[0028] Step Two: Establish a source mapping between the AI content and the final text. Segment the final text at the sentence level to obtain 15 text fragments. Perform the following operations on each text fragment: (1) Using a pre-trained sentence vector model, each text segment and all sentences in the AI-generated responses are mapped to sentence vector representations respectively. The cosine similarity between each text segment and all AI-generated response sentences is calculated as semantic similarity.
[0029] (2) Using the edit distance algorithm (Levenshtein distance), the literal similarity between each text fragment and all AI response sentences is calculated. That is, the edit distance between two strings is divided by the length of the longer string to obtain the normalized literal similarity, which takes values in the range of [0,1].
[0030] (3) The source is classified according to the following rules: If the literal similarity between a text fragment and a piece of AI-generated content is higher than the preset first threshold (set to 0.85 in this embodiment), it is classified as a direct source of AI; if the semantic similarity between the text fragment and a piece of AI-generated content is higher than the preset second threshold (set to 0.75 in this embodiment) but the literal similarity is lower than the first threshold, it is classified as a source of AI rewriting - that is, a combination feature of "semantic similarity but different literal meaning"; if the literal similarity and semantic similarity between the text fragment and all AI-generated content are lower than the corresponding threshold, it is classified as learner original.
[0031] In this embodiment, the source tracing results of the 15 segments are as follows: 5 segments are classified as direct sources of artificial intelligence, 6 segments are classified as sources of artificial intelligence rewriting, and 4 segments are classified as learner-original.
[0032] Step 3: Calculate the cognitive processing depth index. For each of the 11 segments originating from artificial intelligence, combine the source tracing results from Step 2 and the operational data from Step 1 to determine the processing depth: (1) For the five segments directly sourced from artificial intelligence: the corresponding input operation types in the query operation data, three of them are direct paste operations, with the lowest cognitive processing depth, and are judged as surface copying; two are manual re-input operations, with a relatively low cognitive processing depth, and are still judged as surface copying.
[0033] (2) For the six segments from the source of AI rewriting: syntactic dependency analysis was performed on the rewritten segments and the corresponding original AI sentences to extract their respective syntactic structures and detect the degree of change in syntactic structure, information organization order, and argumentation logic of the rewritten segments relative to the original AI text. Four of the segments only underwent word substitution while the syntactic structure and information organization order remained basically unchanged, which was judged as superficial synonym substitution, with a medium cognitive processing depth, and was judged as partial processing; the other two segments underwent substantial changes in syntactic structure or argumentation logic (for example, rewriting the causal structure of "because A, therefore B" in the original AI text into the adversative structure of "although A, but B", or reorganizing the linear argument of the original AI text into a contrastive argument structure), which was judged as structural reconstruction, with a high cognitive processing depth, and was judged as deep processing.
[0034] (3) Among the four learners’ original segments, the cognitive processing depth was the highest, and it was determined to be deep processing.
[0035] Step 4: Generate learning effectiveness assessment results: (1) Proportion of in-depth processing: The ratio of the sum of the word counts of the structurally reconstructed segments (2) and the learner-original segments (4) to the total word count of the full text. In this embodiment, these 6 segments total 580 words, and the full text is 1200 words. The proportion of in-depth processing = 580 / 1200 = 48.3%.
[0036] (2) Percentage of surface copying: The sum of the number of words in the text of directly pasted segments and manually re-entered segments (5 in total). In this embodiment, these 5 segments total 320 words, and the percentage of surface copying = 320 / 1200 = 26.7%.
[0037] (3) Overall score of learning effectiveness: The overall score is obtained by weighting the processing depth of each segment by the number of words. In this embodiment, the directly pasted segment is assigned 0.1 points, the manually re-entered segment is assigned 0.3 points, the superficial synonym replacement segment is assigned 0.5 points, the structural reconstruction segment is assigned 0.8 points, and the learner's original segment is assigned 1.0 points. The scores of each segment are weighted and summed by the number of words and then divided by the total number of words: Overall score = (3×80×0.1+2×80×0.3+4×60×0.5+2×100×0.8+4×145×1.0) / 1200=932 / 1200=0.777.
[0038] (4) Visualized diagnostic report: The final text is marked with different colors on the web page interface - directly pasted fragments are marked in red, manually re-entered fragments are marked in orange, superficially synonymous replacement fragments are marked in yellow, structurally reconstructed fragments are marked in light green, and learner-original fragments are marked in dark green, which intuitively presents the true distribution of learners' contributions for teachers to use in evaluation.
[0039] Example 2
[0040] This example is applied to an academic writing course on an online MOOC platform, requiring learners to complete a research abstract writing task. The platform has an embedded AI writing assistant. In this example, a single assessment batch includes 326 learners.
[0041] Step 1: Collect human-computer interaction data. The platform backend automatically collects dialogue data, operation data, and final text data from all 326 learners via the application programming interface. In this embodiment, the 326 learners generated a total of 4890 sentence-level text fragments.
[0042] Step 2: Establish source mapping. For each text segment, the same sentence vector semantic similarity calculation and edit distance literal similarity calculation methods as in Example 1 are used, and the same threshold is used for source classification as in Example 1. The statistical results are as follows: 1652 segments directly sourced by artificial intelligence (33.8%), 1856 segments rewritten by artificial intelligence (37.9%), and 1382 segments created by learners (28.3%).
[0043] Step 3: Calculate the cognitive processing depth index. For each segment originating from artificial intelligence, the same processing depth determination rules as in Example 1 are used, and the segment is classified by combining operational data and syntactic structure analysis. The statistical results are as follows: direct pasting accounts for 21.5%, manual re-entry accounts for 12.3%, surface synonym replacement accounts for 22.1%, structural reconstruction accounts for 15.8%, and learner originality accounts for 28.3%.
[0044] Step 4: Generate learning effectiveness assessment results. Calculate the percentage of deep processing, the percentage of superficial copying, and the overall learning effectiveness score for each learner. A score of 0.7 or higher is defined as deep learner, 0.4 to 0.7 as intermediate learner, and below 0.4 as superficial learner. In this example, the mean overall score for the 326 learners was 0.583, with a standard deviation of 0.187. Among them, 89 were deep learners (27.3%), 158 were intermediate learners (48.5%), and 79 were superficial learners (24.2%). Teachers can use this classification to implement targeted teaching interventions for learners at different levels.
[0045] Example 3
[0046] This example is applied to a high school Chinese writing teaching scenario. The teacher required students to write an argumentative essay on the topic of "Traditional Culture and Modern Life," and allowed them to use a controlled AI writing assistant provided by the school. A total of 42 students participated.
[0047] Step 1: Collect data on the human-computer interaction process. Using a controlled AI writing assistant and the writing platform's built-in editor, AI dialogue data, text manipulation data, and final text data were collected from each of the 42 students. The students' essays averaged approximately 900 words, and the average number of AI dialogue rounds was 5.3.
[0048] Step 2: Establish source mapping. The same method as in Example 1 is used to perform sentence-level segmentation and source classification for each essay.
[0049] Step 3: Calculate the cognitive processing depth index. The same processing depth determination rules as in Example 1 are used.
[0050] Step 4: Generate learning effectiveness assessment results and formative feedback. In addition to calculating various learning effectiveness indicators, this embodiment also adds a formative feedback function: (1) Generate a personalized cognitive processing diagnostic report for each student, including a full-text source distribution map (showing the proportion of various source fragments) and specific fragment examples (listing 2 to 3 typical copied fragments and 2 to 3 deep-processed fragments respectively) for students to reflect on themselves.
[0051] (2) The teacher adjusts the teaching focus based on the overall class data. In this example, the average comprehensive score of the learning outcomes of the 42 students in the class was 0.621. The teacher found that the class as a whole performed poorly in terms of logical argument reconstruction, with structural reconstruction accounting for only 18.2%. Therefore, the teacher strengthened the special training on argument structure in subsequent courses.
[0052] (3) Longitudinal tracking: This method was continuously applied to multiple writing tasks in the same semester to plot the trend curve of the overall learning performance score of each student. In this embodiment, after continuous tracking of 3 writing tasks, the average overall score of the whole class increased from 0.556 to 0.689, and the proportion of in-depth processing increased from 35.6% to 51.2%, indicating that students gradually improved their behavior of over-reliance on artificial intelligence to copy, and began to process and reconstruct content more independently.
[0053] The embodiments described above are merely illustrative of several implementations of the present invention, and while the descriptions are specific and detailed, they should not be construed as limiting the scope of the present invention. It should be noted that those skilled in the art can make various modifications and improvements without departing from the concept of the present invention, and these modifications and improvements all fall within the scope of protection of the present invention. Therefore, the scope of protection of this patent should be determined by the appended claims.
Claims
1. A method for evaluating learner learning outcomes in human-computer collaborative writing, characterized in that, Includes the following steps: Step 1: Collect data on the human-computer interaction process; During the learner's completion of the AI-assisted writing task, three types of data are automatically collected: AI dialogue data, including the instructions issued by the learner to the AI system and all generated content returned by the AI system; text operation data, including the operation type and operation timestamp when the learner imports the AI-generated content into their own text, the operation type including direct pasting, manual re-entry, and partial editing; and final text data, that is, the final result text submitted by the learner. Step Two: Establish a source mapping between AI content and the final text; compare each text segment in the final text with the AI-generated content to trace its source, classifying it into one of the following categories: direct AI source, i.e., the literal similarity with a certain AI-generated content is higher than a set first threshold; AI rewritten source, i.e., the semantic similarity with a certain AI-generated content is higher than a set second threshold, but the literal similarity is lower than the first threshold, showing semantic similarity but large literal differences; learner original, i.e., there is no obvious correspondence with any AI-generated content; among which, semantic similarity is calculated using semantic representation techniques such as sentence vectors, and literal similarity is calculated using string matching techniques such as edit distance; Step 3: Calculate the cognitive processing depth index; Based on the source tracing results of Step 2 and the operational data of Step 1, calculate the cognitive processing depth score for each text fragment originating from artificial intelligence: For fragments directly from artificial intelligence, the cognitive processing depth is lowest when the operation method is direct pasting, and it is judged as superficial copying; the cognitive processing depth is still low when the operation method is manual re-inputting, and it is still judged as superficial copying; For fragments rewritten by artificial intelligence, by detecting the degree of change of the rewritten fragment relative to the original text of artificial intelligence in syntactic structure, information organization order and argumentation logic, distinguish between superficial synonym replacement that only replaces words and structural reconstruction that has undergone substantial changes. The former has a medium cognitive processing depth and is judged as partial processing, while the latter has a high cognitive processing depth and is judged as deep processing; For fragments created by learners, the cognitive processing depth is the highest and is judged as deep processing; Step 4: Generate learning effectiveness assessment results; comprehensively analyze the processing depth of all text fragments, calculate the proportion of deep processing, the proportion of superficial copying, and the overall learning effectiveness score, and generate a visual diagnostic report, highlighting the true distribution of learners' contributions by marking different source fragments in the final text with different colors.
2. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step one, text operation data is collected by listening to the operating system clipboard event and keyboard input event in the learner's editing interface. For each text import operation, the operation type identifier, operation timestamp, imported text content, and insertion position in the learner's text are recorded.
3. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step two, the semantic similarity is calculated as follows: a pre-trained sentence vector model is used to map text segments into sentence vector representations, and the semantic similarity is obtained by calculating the cosine similarity between two sentence vectors; the literal similarity is calculated as follows: the Levenshtein edit distance algorithm is used to divide the edit distance between two strings by the length of the longer string to obtain the normalized literal similarity.
4. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step two, the source mapping identifies text fragments from which artificial intelligence rewrites them by combining semantically similar but literally different features. That is, fragments with semantic similarity higher than the second threshold but literal similarity lower than the first threshold are classified as artificial intelligence rewrite sources to distinguish them from direct artificial intelligence sources with literal similarity higher than the first threshold.
5. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step three, the method for determining structural reconstruction is as follows: perform syntactic dependency analysis on the rewritten fragment and the original AI text respectively, extract their respective syntactic structures, and detect the degree of change of the rewritten fragment relative to the original AI text in terms of syntactic structure, information organization order, and argumentation logic. If substantial changes occur, it is determined to be structural reconstruction; if only word substitution occurs while the syntactic structure and information organization order remain basically unchanged, it is determined to be superficial synonym substitution.
6. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step four, the deep processing ratio is the proportion of the sum of the word counts of the structurally reconstructed segments and the learner's original segments to the total word count of the full text; the surface copying ratio is the proportion of the sum of the word counts of the directly pasted segments and the manually re-entered segments to the total word count of the full text.
7. The method for evaluating learner learning outcomes in human-computer collaborative writing according to claim 1, characterized in that, In step four, the calculation method for the comprehensive score of learning effectiveness is as follows: each text segment is assigned a score according to the depth of cognitive processing, with direct pasting assigned the lowest score, manual re-entry assigned a lower score, surface synonym replacement assigned a medium score, structural reconstruction assigned a higher score, and learner originality assigned the highest score. The scores of each segment are weighted and summed according to the number of words, and then divided by the total number of words in the whole text to obtain a comprehensive score between 0 and 1.
8. A computer-readable storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the learner learning effectiveness evaluation method in human-computer collaborative writing as described in any one of claims 1 to 7.