Article scoring method based on multidimensional semantic feature deconstruction and weighted fusion

Abstract

The invention relates to the field of large language models, in particular to an article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion, which comprises the following steps: generating an article evaluation system; all specific types of articles are collected from the public document database, and the collected data are divided in proportion according to the scoring dimension by means of a specific type article evaluation system; a multi-classification task model based on a large language model is constructed, qwen3-8b is used as a backbone of the model to extract features from an input article, the features are fused into a weight system in an article evaluation system during feature extraction, and finally, a multi-layer perceptron is used as a classification head to score based on the extracted features; and training the multi-classification task model based on the large language model from the data obtained from the public document database. According to the method, the world knowledge of the sota large language model is fully distilled, the sequence perception distribution cross entropy is put forward to carry out fine adjustment on the large model, and articles can be scored more objectively and accurately.

Description

TECHNICAL FIELD

[0001] The present application relates to the fields of big data technology, deep learning, natural language processing, and large language models, and in particular to an article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion. BACKGROUND

[0002] In recent years, with the explosion of information, text data is growing at an unprecedented rate. Whether it is in the field of education, essay correction, academic paper review, or internet platform content quality control, it has become an urgent need to quickly, accurately and objectively evaluate a large number of articles. Although traditional manual review methods have advantages in judgment depth and accuracy, they are inefficient, costly, and easily affected by factors such as reviewer's subjective preference, knowledge background and fatigue level, resulting in lack of stability and consistency of evaluation results.

[0003] In order to overcome the limitations of manual review, automated article scoring technology has emerged. Early automated methods mostly rely on word frequency, sentence length, spelling errors and other shallow statistical features, which, although to some extent, realize automation, but due to the neglect of the core connotation of the article - semantic information, the scoring results often deviate greatly from the judgment of human experts. They are difficult to effectively measure the logical coherence, argument depth and structural rationality of the article.

[0004] With the rapid development of deep learning and natural language processing technology, it is possible to capture deep semantic information using models. However, existing models often treat articles as a whole when making comprehensive scores, lacking fine-grained analysis of different dimensions of article quality, resulting in a "black box" scoring process that is difficult to explain the specific reasons. SUMMARY

[0005] In order to make the scoring of articles more objective and reliable, the present application proposes an article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion, which specifically includes the following steps:

[0006] S1, based on the thought chain prompt, the large language model is distilled with scoring dimensions and weight systems to generate an article evaluation system, which includes multiple scoring dimensions and the corresponding weights of each dimension;

[0007] S2, collect articles of specific types from public document databases, and use the specific type article evaluation system to divide the collected data proportionally by scoring dimensions, i.e. stratified sampling of the collected data, so that the training set and the validation set maintain the same label proportion, where the label refers to the scoring dimension, which can be theme clarity, article result and logical clarity, language expression fluency, information integrity, etc.

[0008] S3, construct a multi-classification task model based on a large language model, use qwen3-8b as the backbone of the model to extract features from the input article, and integrate the weight system in the article evaluation system when extracting features, and finally use a multi-layer perception as a classification head to score based on the extracted features;

[0009] S4, the data obtained from the public document database is used to train the multi-classification task model based on the large language model, and the trained model is used to score the article.

[0010] The application fully distills the world knowledge of the sota large language model, and creatively proposes sequential perception distribution cross entropy, which can more objectively and accurately score the article. The following several significant beneficial effects can be brought about:

[0011] 1. The world knowledge of the sota large language model is fully distilled in multiple levels, an objective and multi-dimensional scoring system is constructed, and objective weights are assigned to each scoring system, laying a solid foundation for subsequent model training, and fair, objective and reasonable article scores can be output;

[0012] 2. Creatively propose sequential perception distribution cross entropy as the loss function of model training, which effectively solves the possible overfitting problem of the model to the “hard” label, and converts the “hard” single correct answer (such as an article score of 4) into a “soft” probability distribution centered on the correct answer and smoothly decaying to both sides. In this way, when the model's prediction is close to the correct answer, the loss will be small; when the prediction deviates from the correct answer, the loss will increase sharply. Such design enables the model to learn the sequential relationship between labels (such as 0 to 5 is a process from poor to good), and to learn more rich representation information.

[0013] 3. Add MLP as the classification head based on LLM as the backbone, the benefits brought by learning different weights for different tasks are much greater than the single weight processing or LLM direct end-to-end processing method. BRIEF DESCRIPTION OF DRAWINGS

[0014] Figure 1 A flowchart of an article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion according to an embodiment of the application;

[0015] Figure 2 A model structure diagram in an article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion according to an embodiment of the application. DETAILED DESCRIPTION

[0016] With reference to the drawings of the embodiments of the present application, the technical solutions in the embodiments of the present application will be described clearly and completely. Obviously, the described embodiments are only a part of the embodiments of the present application, rather than all the embodiments of the present application. Based on the embodiments of the present application, all other embodiments obtained by a person of ordinary skill in the art without creative work fall within the protection scope of the present application.

[0017] The present application proposes an article scoring method based on multi-dimensional semantic feature disassembly and weighted fusion, such as Figure 1 , specifically comprising the following steps:

[0018] S1, based on the thought chain prompt, the large language model is distilled based on the scoring dimension and the weight system to generate an article evaluation system, and the article evaluation system includes multiple scoring dimensions and the weight corresponding to each dimension;

[0019] S2, collect articles of specific types from a public document database, and divide the collected data by proportion according to the scoring dimensions using the specific type article evaluation system;

[0020] S3, construct a multi-classification task model based on a large language model, use qwen3-8b as the backbone of the model to extract features from the input article, and integrate the weight system in the article evaluation system when extracting the features, and finally use a multi-layer perception as a classification head to score based on the extracted features;

[0021] S4, train the multi-classification task model based on the large language model using the data obtained from the public document database, and use the trained model to score the article.

[0022] The present application proposes an article evaluation system construction method based on a large model, such as Figure 1 , which can be divided into six parts, including constructing an article scoring system, constructing a data set, constructing a multi-classification task model based on LLM, supervised fine-tuning (SFT) of the multi-classification task model, inference on the test set, checking whether the fine-tuning meets the expectations, and finally saving the model parameters when the expectations are met. The model at this time can be used for scoring task of articles to be scored.

[0023] Specifically, the present application adopts the Cot-PE (Chain-of-Thought Prompt Engineering) method, distills world knowledge from the scoring dimension, and generates a set of objective scoring system suitable for multiple types of articles, specifically including:

[0024] S11, By designing a set of Cot-PE-1, distilling the world knowledge of deepseek-r1-think, Qwen3-235B-A22B-think, two versions of scoring dimensions are constructed, including general scoring dimensions and scoring dimensions for different article types, such as argumentative, narrative, and explanatory texts.

[0025] As a specific implementation, this embodiment gives the specific modules of Cot-PE-1, including:

[0026] # Role: You are a top writing theorist and curriculum designer. Your job is to deconstruct the secrets of "good writing" and turn it into a clear, systematic, and operational evaluation system to guide teaching and scoring. Your knowledge system includes a deep understanding of universal principles common to all writing forms, as well as precise grasp of unique evaluation criteria for different text types such as argumentative, narrative, and explanatory texts.

[0027] # Task: Your task is to use all your knowledge to construct a complete, hierarchical article evaluation system through a coherent thought process. This system includes two main parts:

[0028] 1. A set of **general core scoring dimensions** applicable to all types of articles.

[0029] 2. Design a set of **specific supplementary scoring dimensions** for "argumentative", "narrative", and "explanatory" texts, the three main text types.

[0030] # Chain of Thought:

[0031] ### **First stage: Build a general dimension base**

[0032] **Step 1: Define the meta-value of writing**

[0033] First, please think and explain the fundamental value of "a good article" from a philosophical perspective. Regardless of the text type, an excellent text should achieve what in terms of thought, communication and aesthetics? How does it affect the reader?

[0034] **Step 2: Refine the general core dimensions**

[0035] Based on your definition of the meta-value of writing, please deconstruct it into 4 to 6 core, universal, cross-text evaluation dimensions.

[0036] Name each dimension and describe its connotation in one sentence. For example, "depth of thought", "structure and logic", "language quality", etc.

[0037] * Briefly explain why these dimensions are universal and how they collectively support a good article.

[0038] **Step 3: Define ratings and descriptors for common dimensions**

[0039] ### **Phase Two: Constructing Supplementary Dimensions for Specific Text Styles**

[0040] **Step 4: Transition to Stylistic Analysis**

[0041] Now you will enter the realm of specialization. You must recognize that while general dimensions provide a basic framework, evaluating the quality of a specific writing style requires specialized, more targeted "yardsticks." These specific dimensions are used to **supplement** the general dimensions; only by combining the two can a complete evaluation be achieved.

[0042] **Step 5: Analyze specific text types one by one and create supplementary dimensions**

[0043] You will process the following three text types in order: **argumentative essay, narrative essay, and expository essay.**

[0044] For each writing style, please follow these sub-steps:

[0045] * **5.1: Analyze the nature of the genre**: Define the core objective of the genre (e.g., argumentative writing is "persuasion", narrative writing is "empathy", and expository writing is "clarification"), and analyze the key elements for its success.

[0046] * **5.3: Define ratings and descriptors for specific dimensions**: Similar to step 3, set three ratings (Excellent, Good, Needs Improvement) for these new **specific dimensions**, and populate them with specific behavioral descriptors.

[0047] # Output Format, must strictly follow this structure and use Markdown format:

[0048] Please present your final results in two main parts, clearly organized by level. Do not add any extra text before the titles.

[0049] # Part 1: General Article Scoring Dimensions

[0050] | Dimension | Dimension Description | Excellent | Good | Needs Improvement |

[0051] # Part Two: Supplementary Scoring Dimensions for Specific Article Types

[0052] ## 1. Argumentative Essay

[0053] These dimensions should be used in conjunction with general dimensions for a comprehensive evaluation.

[0054] | Dimension | Dimension Description | Excellent | Good | Needs Improvement |

[0055] ## 2. Narrative Writing

[0056] These dimensions should be used in conjunction with general dimensions for a comprehensive evaluation.

[0057] | Dimension | Dimension Description | Excellent | Good | Needs Improvement |

[0058] ## 3. Expository Text

[0059] These dimensions should be used in conjunction with general dimensions for a comprehensive evaluation.

[0060] | Dimension | Dimension Description | Excellent | Good | Needs Improvement |

[0061] Based on the Cot-PE-1 template mentioned above, the large model is used to analyze and process the input article process.

[0062] S12. A Cot-PE-2 model was designed, using a small model (Qwen3-8b), to validate and improve the initial scoring dimensions in S11. By using large-scale models for the more arduous tasks of extraction, analysis, and designing scoring dimensions, and using small-scale models for simpler tasks such as summarization and supplementation, this approach fully leverages the advantages of large-scale models in enriching world knowledge while also utilizing the efficiency, compactness, and portability of small-scale models, resulting in more detailed, objective, and complete scoring dimensions.

[0063] As a specific implementation method, this embodiment provides a specific module of Cot-PE-2, including:

[0064] # Role: You are an expert with extensive experience in **performance appraisal, educational measurement, and standards design**. Your core competency is building clear, fair, comprehensive, and workable evaluation systems. You have rigorous thinking, adhere to the MECE principle (mutually exclusive, collectively exhaustive), and are adept at identifying potential logical flaws, ambiguities, and areas for improvement.

[0065] # Task: Your task is to receive two sets of scoring dimensions for the **writing quality of an article**, and to critically review, verify, and refine them. Ultimately, you need to output a detailed analysis report and a more scientifically sound set of scoring dimensions.

[0066] # Original rating dimensions:

[0067] [Rating Dimension 1 (generated by deepseek-r1-think), Rating Dimension 2 (generated by Qwen3-235B-A22B-think)]

[0068] # Chain of Thought: Please strictly follow the following thought process steps in your analysis, and clearly demonstrate each step of your thought process in your final answer:

[0069] **Step 1: Understand the core objectives and assessment context.**

[0070] First, carefully read the "Assessment Subjects" and "Original Rating Dimensions" I provided.

[0071] * Reflection: What is the ultimate purpose of this assessment? (For example: Is it to provide formative feedback to students? Or to conduct selective ranking? Or to evaluate employee performance?)

[0072] Who are the users of the evaluation? (e.g., teachers, competition judges, or department managers?)

[0073] Given this goal and background, are the current scoring dimensions correct in direction?

[0074] **Step Two: Analyze the "atomic properties" of each dimension one by one.**

[0075] **Clarity:** Are the names and descriptions for this dimension clear and unambiguous? Could an untrained assessor accurately understand their meaning? Are there any vague terms (such as "relatively good," "somewhat in-depth")?

[0076] * **Measurability:** Can this dimension be objectively observed and measured? Or is it too subjective, easily leading to huge differences in scores among different evaluators? Are the scoring criteria specific and operational?

[0077] * **Singleness:** Does this dimension measure only a single, independent trait? Or does it mix multiple different concepts together? (For example, "creativity and practicality" combines two dimensions and should be separated).

[0078] **Step 3: Evaluate the overall structure of the dimensional system (MECE principle).**

[0079] * **Independent:** Are there significant conceptual overlaps between the dimensions? Is it possible that the evaluator, while evaluating dimension A, actually double-checked parts of dimension B? If overlaps exist, please specify which dimensions are involved and consider how to resolve them through merging, splitting, or redefining.

[0080] * **Complete Exhaustive:** Does this dimensional system omit any key evaluation aspects relative to the assessment objectives? Are there any "weakest links" that haven't been considered? If so, which dimensions should be added?

[0081] **Step 4: Review the scoring criteria and weightings.**

[0082] * **Leveling Reasonableness:** Are the score levels (e.g., 1-5) reasonable? Does the description of each score level clearly distinguish the differences in performance level? Is there a logical progression in the descriptions from low to high scores?

[0083] * **Weight Logic:** (If weights exist) Do the weight allocations for each dimension reflect their importance to the overall goal? For example, for a scientific paper, should the weight of "rigor of argumentation" be higher than that of "elegance of language"? Does the current weight allocation need to be adjusted?

[0084] **Step 5: Formulate specific optimization suggestions.**

[0085] Based on the analysis in steps 1-4 above, all the problems of the original dimensions are systematically summarized.

[0086] For each problem, provide specific and actionable modification suggestions. For example:

[0087] * “It is recommended to change the name of dimension 'A' from 'XX' to 'YY', because 'YY' is more accurate.”

[0088] * "It is recommended to merge dimensions 'B' and 'C' into a new dimension 'D' because they have significant overlap. The description of the new dimension 'D' is as follows..."

[0089] * "It is recommended to add a new dimension 'E: [Dimension Name]' because the original system neglected to consider... Its scoring criteria are as follows..."

[0090] * “It is recommended to change the description of the 3-point criteria in dimension 'F' from '...' to '...' to make it more operational.”

[0091] # Output format requirements

[0092] Please organize your final answer according to the following two parts:

[0093] **Part One: The Analysis and Reasoning Process**

[0094] Please clearly demonstrate each step of your thought process following the above "chain of thought instructions." For example:

[0095] * **1. Core Objective Analysis:** [Your Analysis Content]

[0096] * **2. Analysis of Atomic Properties in Various Dimensions:**

[0097] * Dimension A: [Clarity, Measurability, Uniqueness Analysis]

[0098] * Dimension B: [Clarity, Measurability, Uniqueness Analysis]

[0099] * ...

[0100] * **3. MECE Analysis:** [Mutual Exclusivity and Completeness Analysis]

[0101] * **4. Scoring Criteria and Weighting Analysis:** [Grading and Weighting Analysis]

[0102] * **5. Summary of Optimization Suggestions:** [Systematically lists all suggested modifications]

[0103] **Part Two: Revised Scoring Dimensions (Final Version)**

[0104] * After completing all analyses and recommendations, please provide a complete, clean, and ready-to-use "final version" rating scale.

[0105] * It is recommended to use Markdown Table format, which includes **dimension name, dimension description, weight (optional), and specific criteria for each score level**.

[0106] S13. Design a Cot-PE-3 model, adding deepseek-r1-think and Qwen3-235B-A22B-think respectively. Weight the scoring dimensions obtained in S12, and finally, take the average of the two models for the weight of each scoring dimension. This approach balances the strengths of both models and fully utilizes their respective world knowledge.

[0107] As a specific implementation method, this embodiment provides a specific module of Cot-PE-3, including:

[0108] # Role: You are a seasoned **strategic analyst** and **assessment system design expert**. Your core competency lies in objectively assessing the relative importance of different components within a complex system to ensure that the final assessment accurately reflects its core objectives. You excel at value judgments and prioritization, and can logically justify each of your decisions.

[0109] # Task: Your task is to assign weights to a set of finalized rating dimensions. You need to assign a reasonable percentage of weight to each dimension based on the given **[core evaluation objectives]**.

[0110] **Core Requirements:**

[0111] 1. The sum of the weights of all dimensions must be exactly equal to 100%.

[0112] 2. The allocation process cannot rely solely on intuition; it must involve clear and traceable logical reasoning.

[0113] 3. The final weighting scheme must reflect a deep understanding of the evaluation objectives and be **fair, reasonable, and focused**.

[0114] # Input information:

[0115] **1. Evaluation Core Objective: To provide the fairest and most reasonable weight allocation scheme.**

[0116] **2. Final Version Rating Dimensions:**

[0117] [Small model summarizing and refining scoring dimensions]

[0118] # Chain of Thought

[0119] Please strictly follow the thought process outlined below to complete the weight allocation task, and clearly demonstrate your thought process at each step in your final answer:

[0120] **Step 1: Deeply understand the core objectives and establish "value anchors".**

[0121] First, please restate and analyze in detail the "core objectives of the assessment" I provided.

[0122] * Think and answer: To achieve this goal, what is the **most critical and indispensable** quality of the evaluated object? If you could only summarize it in one word or sentence, what would it be? This quality is the "value anchor" or "North Star indicator" for your weighting.

[0123] **Step 2: Classify and sort the dimensions**

[0124] * Based on the "value anchors" established in the first step, all rating dimensions are categorized into different importance levels. A three-level classification method is recommended:

[0125] ****First Tier (Core Foundation):** Without outstanding performance in these dimensions, the object being evaluated will fundamentally lose its value. They directly determine the success or failure of the objective.

[0126] * **Second Tier (Important Support):** These dimensions greatly influence the quality and effectiveness of the evaluated object and are an important guarantee for achieving the goal.

[0127] * **Third Tier (Auxiliary Optimization):** These dimensions are bonus points or basic requirements that can make the evaluated object more complete, but they are not decisive factors.

[0128] Please clearly list the dimensions included in each level and briefly explain your reasoning for this division.

[0129] **Step 3:** Employ a top-down strategy to allocate total weight across all levels.

[0130] Based on the importance of the three levels, an overall weight range is assigned to each **level**. This is a macro-level resource allocation process.

[0131] For example, you can decide:

[0132] The first tier (core foundation) accounts for 60% - 70% of the total weight.

[0133] The second level (important support) accounts for 20% - 30% of the total weight.

[0134] The third level (auxiliary optimization) accounts for a total weight of 5% - 10%.

[0135] Please explain why you allocated the total weight of the levels in this way, and how this reflects the focus on the core objectives.

[0136] **Step 4:** Fine-tune the hierarchy from the bottom up to determine the final weights.

[0137] Now, assign specific weights within each level.

[0138] **For dimensions within a hierarchy:** Are they all equally important? If not, which is more important? Why?

[0139] * Refine the total weights you allocated in step three to each dimension within that level, following the considerations outlined above.

[0140] * Ensure that the sum of all dimension weights is exactly equal to 100%. If the initial allocation is not equal to 100%, please make fine adjustments and explain the logic behind the adjustments (e.g., from which dimension's weights to increase / decrease, and why).

[0141] **Step 5: Conduct a final review and justification.**

[0142] * Review your complete weighting scheme.

[0143] * Conduct a thought experiment: Suppose there are two evaluation targets, one scoring highly on the "Core Foundation" dimension but poorly on the "Auxiliary Optimization" dimension, and the other the opposite. With your designed weights, will the former's total score be significantly higher than the latter's? Does this align with the initial evaluation objective?

[0144] Finally, write a concluding statement that systematically explains why your weighting scheme is the **fairest and most reasonable**.

[0145] # Output format requirements

[0146] Please organize your final answer according to the following two parts:

[0147] **Part 1: The Reasoning Process for Weight Allocation**

[0148] * Please strictly follow the five steps of the above "Thinking Chain Instructions" to clearly demonstrate your entire thinking process.

[0149] * **1. Core Objectives and "Value Anchors":** [Your Analysis]

[0150] * **2. Dimensional Classification and Sorting:** [Your classification list and reasons]

[0151] * **3. Overall Weight Allocation by Level:** [Your macro-level allocation plan and rationale]

[0152] * **4. Internal Weight Fine-tuning:** [Your fine-tuning process and calculations]

[0153] **5. Final Review and Argumentation:** [Your thought experiment and final argument]

[0154] **Part Two: Final Weight Allocation Scheme**

[0155] Use a clear Markdown table to present the final result.

[0156] | Dimension Name | Hierarchical Level | Weight (%) | Reason for Allocation |

[0157] | [Dimension A Name] | [Example: First Level] | [XX] | [A concise explanation of why this weight is used] |

[0158] | [Dimension B Name] | [Example: Second Level] | [YY] | [A concise explanation of why this weight is used] |

[0159] | ... | ... | ... | ... |

[0160] | **Total** | - | **100%** | - |

[0161] A total of 8,000 articles were collected from public document databases such as Google Scholar and Baidu Wenku. The aforementioned scoring system was used to annotate these 8,000 articles, with an annotation ratio of 70% LLM annotation and 30% manual annotation. The dataset was then divided according to the scoring dimensions obtained in S11-S13, with 6,000 articles in the training set and 2,000 articles in the validation set. If any dataset was insufficient, 1,000 new articles were collected, LLM-annotated, and added to the corresponding dataset. The specific steps are as follows:

[0162] S21. Use the scoring dimensions of S11~S13 to annotate 8000 data points. 70% of the data is annotated using the Cot-PE method with LLM (deepseek-r1-think); the remaining 30% of the data is annotated manually.

[0163] S21. The labeled data is divided proportionally according to the scoring dimensions obtained in S11~S13, with 6000 data points in the training set and 2000 data points in the validation set. After division, if one class of data is insufficient, such as the number of argumentative papers in the training set not meeting the requirement, 1000 new data points are collected for LLM annotation, and the argumentative papers are used to supplement the missing parts of the training set. Finally, the LLM annotation results and the manual annotation results need to be sampled and checked separately. If the bad case rate is >= 5%, Cot-PE needs to be redesigned for annotation or manual re-annotation is required. By combining LLM annotation and manual annotation, the disadvantages of subjective annotation results that may be generated by manual annotation are effectively avoided, while taking advantage of the advantages of LLM in enriching world knowledge, objectivity, and efficiency.

[0164] Construct a multi-class classification task model based on LLM. Use qwen3-8b as the model backbone. Based on the S1 scoring criteria, add an MLP as the classification head. The data processing procedure is as follows. Figure 2 ,include:

[0165] S31. Tokenize the input text to convert it into a Qwen model that can understand the sequence of numeric IDs. .

[0166] S32, Read the numeric ID sequence It also generates a high-dimensional vector representation rich in contextual information for each token. ,in , Represents the set of real numbers. L is the hidden layer dimension of the model, and L is the number of tokens in the input text. For autoregressive models like qwen, the hidden state of the last token typically contains a summary of all the preceding content, integrating information from all previous layers and the context of the entire sequence. This generates the deepest and richest understanding of each token, making it ideal for classification tasks. Therefore, the hidden state of the last token is chosen as the representation of the entire sequence for downstream tasks.

[0167]

[0168] S33. Average pooling is performed on the hidden states of the last layer, and the word-level representations of each sentence are aggregated into sentence-level representations, which are used to match the data representations of downstream tasks.

[0169] S34. Input the pooled hidden layer state into the MLP to obtain information for each category. The formula is as follows:

[0170]

[0171]

[0172] in, The output classification matrix of the multilayer perceptron has dimensions (batch_size, num_class), where batch_size is the size of a training batch of data. For classification matrix A vector consisting of the elements in the j-th column; This represents the activation function, which takes... and The maximum value is expressed as ; This represents the sentence-level representation of the word segmentation sequence after average pooling, with dimensions (batch_size, hidden_size), where hidden_size is the dimension of the hidden layer in the qwen3-8b model; W1 is the learnable weight matrix. Dimensions , Set to 512; , Dimensions , This indicates the number of scoring dimensions in the article scoring system; express Each vector in probability , which is the predicted score of the j-th rating dimension.

[0173] S35. During training, this invention creatively proposes Ordinal-Aware Distributive Cross-Entropy (OADC) as the loss function. This is because when rating each dimension, assuming a score of 1-5, where 1 represents very poor and 5 represents the best, simply labeling the model with 1-5 will only teach it that one article corresponds to one score, failing to learn how to differentiate between good and bad articles. Therefore, this invention creatively proposes the Ordinal-Aware Distributive Cross-Entropy function. Its core idea is to transform the "hard" single correct answer (e.g., an article's score of 4) into a "soft" probability distribution centered on the correct answer and smoothly decreasing outwards. This way, when the model's prediction is close to the correct answer, the loss is small; when the prediction is far from the correct answer, the loss increases dramatically. The formula for Ordinal-Aware Distributive Cross-Entropy is as follows:

[0174]

[0175] The core calculation of the formula consists of two parts, mainly the probability distribution of soft targets. Calculate and predict probability distribution Calculation. The formula for calculating the probability distribution of soft targets is as follows:

[0176]

[0177] The formula for calculating the prediction probability is as follows:

[0178]

[0179] The variables defined in the loss function include: This is the training batch size, i.e., the number of samples trained in each session. C represents the number of categories, i.e., the number of ratings. For example, if the rating is 1-5 stars, C=5. The index of the sample in the batch, from 1 to N; Index for the category, from 0 to C; Let C be the true integer label of the i-th sample, which is a value between 0 and C-1; Let logits be the original output of the model for the i-th sample, which is a vector of length C, i.e. ; The standard deviation of the Gaussian distribution is a hyperparameter used to control the "smoothness" or "sharpness" of the soft label distribution. The soft target probability distribution generated for the i-th sample is a vector of length C, i.e. ; Let the model predict the probability distribution of the i-th sample, by... The result obtained by applying the AccMax function is represented as follows: ; This represents the final calculated total loss for the batch.

[0180] S4. The qwen3-8b model used in this invention is a general-purpose base model. This model itself does not use the scoring task specific to this invention's scenario. Therefore, this invention uses the constructed training data to fine-tune this model to adapt it to the feature extraction of the multi-classification task of this invention. Specifically, to save GPU memory, this embodiment considers using LoRa for fine-tuning, where the key parameters for fine-tuning are as follows:

[0181] batch_size=32

[0182] lr=1e-5

[0183] epoch=5

[0184] lora_rank=16

[0185] Where batch_size represents the size of each training batch during fine-tuning; learning rate represents the model learning rate during fine-tuning; Epoch represents the number of training epochs of the model during fine-tuning; and Lora_rank represents the dimension of the low-rank decomposition matrix during the current process of training the model with Lora during fine-tuning.

[0186] S5. Perform inference on the test set and check whether it meets expectations. Solve for F1 by comparing the values ​​of the model results and the manually labeled results. If F1 < 0.9, return to step S4; otherwise, proceed to step S6.

[0187] S6. The model performance meets expectations; save the model weights. Use the saved model parameters to score the articles to be rated.

[0188] Although embodiments of the invention have been shown and described, it will be understood by those skilled in the art that various changes, modifications, substitutions and alterations can be made to these embodiments without departing from the principles and spirit of the invention, the scope of which is defined by the appended claims and their equivalents.

Claims

1. An article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion, characterized in that, Specifically, the following steps are included: S1. Based on the thought chain prompts, the process of the large language model is distilled with scoring dimensions and weight system to generate an article evaluation system. The article evaluation system includes multiple scoring dimensions and the corresponding weight of each dimension. S2. Collect articles of various specific types from public document databases, and divide the collected data according to the scoring dimensions using the evaluation system for specific types of articles. S3. Construct a multi-classification task model based on a large language model. Use qwen3-8b as the backbone of the model to extract features from the input article. When extracting features, incorporate the weight system in the article evaluation system. Finally, use a multilayer perceptron as the classification head to score based on the extracted features. S4. Data obtained from public document databases is used to train a multi-classification task model based on a large language model, and the trained model is used to score articles.

2. The article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion according to claim 1, characterized in that, The scoring process of a multi-classification task model based on a large language model includes: The article is segmented into words, which is then converted into a word sequence. The word sequence is then embedded and encoded into a sequence of numeric IDs. The numeric ID sequence is used as input to the qwen3-8b model, which outputs the contextual representation of each word in the article and uses the hidden state of the last word as the sentence-level representation of the word segmentation sequence. The sentence-level representation of the word segmentation sequence is averaged and then used as input to a multilayer perceptron. The multilayer perceptron outputs the score of the article on each scoring dimension, and the average of all scores is used as the article's score.

3. The article scoring method based on multidimensional semantic feature deconstruction and weighted fusion according to claim 2, characterized in that, The classification process of a multilayer perceptron includes: in, The output classification matrix of the multilayer perceptron has dimensions (batch_size, num_class), where batch_size is the size of a training batch of data. For classification matrix A vector consisting of the elements in the j-th column; Represents the activation function, taking and The maximum value, that is: ; To perform sentence-level representation of the segmented word sequence after average pooling, its dimension is (batch_size, hidden_size), where hidden_size is the dimension of the hidden layer in the qwen3-8b model; ; , It is a learnable bias vector; This indicates the number of rating dimensions.

4. The article scoring method based on multi-dimensional semantic feature deconstruction and weighted fusion according to claim 1, characterized in that, We propose a sequence-aware distributed cross-entropy function to train a multi-classification task model based on a large language model, namely: in, is the cross-entropy function of the sequence-aware distribution; N is the training batch size, i.e., the number of samples in each training iteration; C is the number of categories to be classified. Let be the probability distribution of the i-th sample belonging to category c of soft targets; Let be the predicted probability distribution of the i-th sample belonging to category c.

5. The article scoring method based on multidimensional semantic feature deconstruction and weighted fusion according to claim 4, characterized in that, The probability distribution of the i-th sample belonging to category c of soft targets Represented as: in, Let i be the true class label of the i-th sample; Hyperparameters for controlling the distribution of soft tags.

6. The article scoring method based on multidimensional semantic feature deconstruction and weighted fusion according to claim 4, characterized in that, The predicted probability that the i-th sample belongs to class c Represented as: in, Output the probability that the i-th sample belongs to category c for a multi-class classification task model based on a large language model.

7. The article scoring method based on multidimensional semantic feature deconstruction and weighted fusion according to claim 1, characterized in that, The trained multi-class classification task model based on a large language model scores the data in the test set, and the F1 score is calculated. If the F1 score is less than 0.9, the LoRa method is used to fine-tune the trained multi-class classification task model based on the large language model. The fine-tuning parameters include: batch_size=32; lr=1e-5； epoch=5; lora_rank=16; Where batch_size represents the size of a batch of training data; learning rate represents the model learning rate during fine-tuning; Epoch represents the number of training epochs of the model during fine-tuning; and Lora_rank represents the dimension of the low-rank decomposition matrix during the current Lora training process.

8. The article scoring method based on multidimensional semantic feature deconstruction and weighted fusion according to claim 1 or 3, characterized in that, The process of generating an article evaluation system includes: Let at least two existing large language models define the meta-values ​​of general writing respectively, and extract multiple cores from the defined meta-values ​​as evaluation dimensions, and provide evaluation levels and descriptors for each evaluation dimension; Based on the general evaluation, at least for argumentative, narrative and expository texts, core objectives in addition to the general core are defined. Based on the core objectives of specific text types, corresponding evaluation dimensions are supplemented, and evaluation levels and descriptors are provided for each supplemented evaluation dimension. By combining general evaluation descriptions with evaluation descriptions of characteristic text styles, a preliminary article evaluation system is formed. Merge all article evaluation systems built by large language models, and use a smaller large language model to check and correct the article evaluation systems built by at least two large language models. The check and correction include merging the same evaluation dimensions and deleting unreasonable evaluation dimensions. By using multiple large language models to assign weights to each evaluation dimension in the revised article evaluation system, and using the average of the weights assigned by multiple models as the final weight of that evaluation dimension, the article evaluation system is completed.

Images