Automatic evaluation method for evaluation data by semantic recognition model based on machine learning

Abstract

The invention discloses a method for automatically evaluating evaluation data through a semantic recognition model based on machine learning, and the method comprises the following steps: S1, receiving an evaluation filling text and scale configuration data, carrying out the preprocessing of the filling text, and forming a standardized evaluation text; s2, calling LLaMA to perform semantic analysis and splitting on the standardized evaluation text to generate a semantic unit text; s3, constructing a scoring anchor point library according to the scale configuration data; s4, performing vector coding on the semantic unit text and the scoring anchor point text to obtain uniform vector representation; s5, generating a score hypothesis set corresponding to the semantic unit based on the vector similarity; s6, performing multi-layer contrast verification on the score hypothesis, and screening to obtain the score hypothesis passing the verification; and S7, summarizing semantic unit scoring results, calculating dimension scores according to scale dimensions, and combining weights and reverse scoring to obtain a total evaluation score. According to the method, automatic semantic splitting and stable scoring of the evaluation text are realized, and the evaluation efficiency and the result consistency are improved.

Description

Technical Field

[0001] This invention relates to the field of natural language processing technology, and in particular to an automatic evaluation method for assessment data based on a machine learning-based semantic recognition model. Background Technology

[0002] In current assessment processes, questionnaires are typically completed online or offline by respondents and submitted to a backend system. The backend then exports the assessment data in Excel or Word format for measurement personnel to process. For questionnaires containing open-ended text, subjective descriptions, or supplementary explanations, measurement personnel need to read each item in the exported table or document, first categorizing the text by dimension, then breaking down the content, performing semantic analysis, and scoring accordingly. This process relies on human understanding and experience, resulting in long processing cycles, repetitive work, and difficulty in adapting to application scenarios involving large-scale data aggregation.

[0003] In existing technologies, some solutions use keyword matching, rule templates, or simple similarity calculations to achieve automatic scoring. However, these methods often only process surface-level text features and lack the ability to recognize continuous sentences, implicit semantics, negation and transitions, and multi-topic expressions. This can easily lead to inaccurate dimension assignment and unstable level classification. Scoring rules usually require frequent manual maintenance, and the transfer cost is high when facing different scales or different groups' expression habits. In collaborative scenarios, manual scoring inevitably suffers from semantic understanding biases and inconsistencies in scale, resulting in score differences for the same text among different measurers, affecting the comparability and consistency of results.

[0004] Therefore, how to provide an automatic evaluation method for assessment data based on machine learning semantic recognition models is a problem that urgently needs to be solved by those skilled in the art. Summary of the Invention

[0005] One objective of this invention is to propose an automatic evaluation method for assessment data based on a machine learning-based semantic recognition model. This invention utilizes a large language model and a sentence vector model to perform semantic analysis and semantic segmentation on the assessment content, and combines the scale structure to complete automatic scoring and summary processing, thereby realizing the intelligent generation of assessment results. It has the advantages of high automation, strong semantic understanding consistency, and high scoring efficiency.

[0006] The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to an embodiment of the present invention includes the following steps:

[0007] S1. Receive the assessment text and scale configuration data, perform preprocessing on the assessment text to obtain standardized assessment text, and the scale configuration data package contains scale dimensions, scale levels, dimension weights and reverse scoring tags.

[0008] S2. Call LLaMA to perform semantic parsing on the standardized evaluation text, identify semantic topics and split them according to semantic boundaries to generate semantic unit text;

[0009] S3. Read the scale configuration data to generate a scoring anchor library, and use the set of control anchors to organize and verify the scale dimensions and scale levels.

[0010] S4. Call the SBERT sentence vector model to perform vector encoding on the semantic unit text to obtain a set of semantic unit vectors, and perform vector encoding on the anchor text of the scoring anchor library to obtain a set of anchor vectors.

[0011] S5. Perform similarity retrieval on each semantic unit vector to obtain candidate anchor points, and generate a set of rating hypotheses based on the scale dimension and scale level corresponding to the candidate anchor points.

[0012] S6. For each scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the control anchor set, and compare the similarity statistics to determine the semantic unit scoring result.

[0013] S7. Summarize the semantic unit scoring results and aggregate them by scale dimension to obtain dimension scores. Convert the dimension scores according to dimension weights and reverse scoring marks to obtain the total score, and output the dimension scores and total scores.

[0014] Optionally, S2 specifically includes:

[0015] S21. Call the LLaMA large language model to receive the standardized assessment text, perform overall semantic parsing on the standardized assessment text, and identify the topic semantics, behavioral semantics, emotional semantics and cognitive semantics contained in the standardized assessment text;

[0016] S22. Based on the semantic parsing results, detect the semantic transfer position and semantic integrity boundary in the standardized evaluation text, and divide the standardized evaluation text into several mutually distinguishable semantic unit texts, each semantic unit text corresponding to a natural language content.

[0017] S23. Number the generated semantic unit texts to form a semantic unit text set.

[0018] Optionally, S3 specifically includes:

[0019] S31. Read the scale dimension field and scale level field from the scale configuration data, and for each scale dimension and each scale level, obtain the text description data corresponding to the scale dimension and scale level.

[0020] S32. Associate the text description data with the corresponding scale dimension identifiers and scale level identifiers one by one to generate a rating anchor record that includes scale dimension identifiers, scale level identifiers and text description content.

[0021] S33. Traverse all rating anchor records and divide the rating anchor records with the same scale dimension identifier and consecutive scale level identifiers into adjacent anchor record groups.

[0022] S34. Mark the rating anchor records, divide the rating anchor records with different scale dimensions into non-same dimension anchor record groups, and divide the rating anchor records with different scale levels into non-same level anchor record groups.

[0023] S35. Store the scoring anchor record, adjacent anchor record group, non-same dimension anchor record group, and non-same level anchor record group to generate a scoring anchor library.

[0024] Optionally, S4 specifically includes:

[0025] S41. Call the SBERT sentence vector model to perform vector encoding processing on each semantic unit text to generate semantic unit vectors corresponding to each semantic unit text.

[0026] S42. Call the SBERT sentence vector model to perform vector encoding processing on the text description content in the scoring anchor record one by one to generate anchor vectors corresponding to each scoring anchor record.

[0027] S43. Perform vector format unification processing on semantic unit vectors and anchor vectors. Vector format unification processing includes vector dimension consistency verification, vector numerical type unification, and vector normalization processing.

[0028] Optionally, S5 specifically includes:

[0029] S51. For each semantic unit vector, calculate the vector similarity between the semantic unit vector and all anchor vectors.

[0030] S52. Sort the vector similarity results corresponding to each semantic unit vector according to the numerical value, and select anchor vectors whose similarity values ​​are within a predetermined range as candidate anchors.

[0031] S53. Read the scale dimension identifier and scale level identifier corresponding to the candidate anchor point, and combine the scale dimension identifier and scale level identifier contained in the candidate anchor point corresponding to the same semantic unit vector to generate a set of rating hypotheses. Each rating hypothesis consists of a set of scale dimension identifier and scale level identifier.

[0032] S54. Perform deduplication on the set of rating hypotheses, delete rating hypotheses whose scale dimension identifiers and scale level identifiers are exactly the same, retain unique rating hypotheses, and generate a set of rating hypotheses corresponding to semantic units.

[0033] Optionally, S5 specifically includes:

[0034] S61. For each rating hypothesis corresponding to the same semantic unit, calculate the similarity statistic between the semantic unit vector and the anchor vector in the rating anchor record corresponding to the rating hypothesis, and obtain the target similarity value of the rating hypothesis.

[0035] S62. Compare the target similarity value with the first threshold. When the target similarity value is less than the first threshold, terminate the verification process of the scoring hypothesis.

[0036] S63. When the target similarity value is greater than or equal to the first threshold, calculate the similarity statistic between the semantic unit vector and the anchor vector in the adjacent anchor record group. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value is within the preset level tolerance range, retain the current scoring hypothesis and record the adjacent levels.

[0037] S64. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value exceeds the preset level tolerance range, calculate the similarity statistic between the semantic unit vector and the anchor vector in the non-same level anchor record group. When the similarity statistic corresponding to the non-same level anchor record group is greater than the target similarity value, terminate the current scoring hypothesis and generate a new level scoring hypothesis.

[0038] S65. When the similarity statistic corresponding to the anchor record group of different levels is less than or equal to the target similarity value, calculate the similarity statistic between the semantic unit vector and the anchor vector in the anchor record group of different dimensions.

[0039] S66. When the similarity statistic corresponding to the non-same dimension anchor point record group is greater than the target similarity value, terminate the scoring hypothesis under the current scale dimension and transfer the semantic unit to the scoring process corresponding to the non-same dimension.

[0040] S67. When the target similarity value is greater than the similarity statistics corresponding to the adjacent anchor record group, the non-same level anchor record group, and the non-same dimension anchor record group, the current scoring hypothesis is determined as the scoring hypothesis that passes the screening.

[0041] Optionally, the scoring process described in S66 specifically includes:

[0042] S661. For the same semantic unit, read the set of rating hypotheses that have passed the filtering, and extract the scale dimension identifier and scale level identifier corresponding to each rating hypothesis.

[0043] S662. When the set of rating hypotheses that have passed the screening is empty, the semantic unit is marked as a low-confidence semantic unit and the output of the rating result of the semantic unit is terminated.

[0044] S663. When the set of rating hypotheses that has passed the screening is not empty, determine whether the scale level identifiers in the set of rating hypotheses are the same.

[0045] S664. When all scale level identifiers in the rating hypothesis set are the same, the scale level identifier is used as the level result of the semantic unit, and the corresponding scale dimension identifier is used as the dimension result of the semantic unit.

[0046] S665. For the rating hypotheses corresponding to different scale levels in the rating hypothesis set, compare the target similarity with the target similarity of the rating hypotheses corresponding to adjacent scale levels respectively; when the target similarity of a rating hypothesis corresponding to a certain scale level is greater than the target similarity of the rating hypotheses corresponding to adjacent scale levels, select the scale level as the rating result of the semantic unit.

[0047] S666. When all rating hypotheses corresponding to the scale levels in the rating hypothesis set fail to be validated, the semantic unit is marked as a low-confidence semantic unit, and the output of the rating result of the semantic unit is terminated.

[0048] S667. Output the scale dimension identifier and scale level result of the semantic unit as the scoring result of the semantic unit.

[0049] Optionally, S7 specifically includes:

[0050] S71. Collect the semantic units with determined rating results according to the scale dimension identifiers, and extract the scale level results of the corresponding semantic units for each scale dimension identifier.

[0051] S72. For each scale dimension identifier, perform numerical summary processing on the collected scale level results to obtain the dimension score corresponding to the scale dimension identifier.

[0052] S73. Read the dimension weights and reverse scoring markers in the scale configuration data, perform reverse conversion and weighted calculation on the dimension scores corresponding to each scale dimension identifier, summarize to obtain the total evaluation score, and output the dimension scores corresponding to each scale dimension identifier and the total evaluation score.

[0053] The beneficial effects of this invention are:

[0054] (1) This invention introduces a large language model to perform semantic parsing and semantic decomposition on the assessment text, transforming the original manual text classification, content decomposition and semantic understanding process into an automatically executable calculation process, realizing the structured processing of the assessment text from the whole to the semantic unit, reducing the degree of manual participation of measurement personnel in the data sorting and preliminary analysis stage, improving the assessment processing efficiency, and is suitable for centralized processing scenarios of large-scale assessment data.

[0055] (2) This invention combines sentence vector model and scoring anchor library to construct scoring hypothesis and verification mechanism. During the scoring process, it introduces adjacent level, non-same level and non-same dimension comparison anchors for multi-level verification, avoids directly scoring based on a single similarity result, reduces the impact of semantic boundary ambiguity, level confusion or dimension misjudgment on the scoring result from the process, improves the stability and consistency of the scoring process, and reduces the error caused by semantic understanding differences when multiple people participate in the scoring.

[0056] (3) The present invention sets up a clear verification failure and rollback processing mechanism in the process of determining the scoring results. When the semantic unit cannot form a stable scoring conclusion, it can mark and terminate the output to avoid unreliable scoring results from entering the subsequent summary calculation. This improves the credibility and interpretability of the evaluation results as a whole, and at the same time makes the scoring process have good robustness and scalability, which is convenient for promotion and application in different scales and different application scenarios. Attached Figure Description

[0057] The accompanying drawings are provided to further illustrate the invention and form part of the specification. They are used in conjunction with embodiments of the invention to explain the invention and do not constitute a limitation thereof. In the drawings:

[0058] Figure 1 This is a flowchart of the automatic evaluation method for assessment data based on a machine learning-based semantic recognition model proposed in this invention.

[0059] Figure 2 This is a diagram showing the construction of the scoring anchor point library and the anchor point grouping structure of the automatic evaluation method for assessment data based on the semantic recognition model of machine learning proposed in this invention.

[0060] Figure 3 This diagram illustrates the semantic unit scoring determination and backtracking process of the automatic evaluation method for assessment data based on the machine learning-based semantic recognition model proposed in this invention. Detailed Implementation

[0061] The present invention will now be described in further detail with reference to the accompanying drawings. These drawings are simplified schematic diagrams, illustrating only the basic structure of the invention, and therefore only show the components relevant to the invention.

[0062] refer to Figure 1-3 An automatic evaluation method for assessment data based on a machine learning-based semantic recognition model includes the following steps:

[0063] S1. Receive the assessment text and scale configuration data, perform preprocessing on the assessment text to obtain standardized assessment text, and the scale configuration data package contains scale dimensions, scale levels, dimension weights and reverse scoring tags.

[0064] S2. Call LLaMA to perform semantic parsing on the standardized evaluation text, identify semantic topics and split them according to semantic boundaries to generate semantic unit text;

[0065] S3. Read the scale configuration data to generate a scoring anchor library, and use the set of control anchors to organize and verify the scale dimensions and scale levels.

[0066] S4. Call the SBERT sentence vector model to perform vector encoding on the semantic unit text to obtain a set of semantic unit vectors, and perform vector encoding on the anchor text of the scoring anchor library to obtain a set of anchor vectors.

[0067] S5. Perform similarity retrieval on each semantic unit vector to obtain candidate anchor points, and generate a set of rating hypotheses based on the scale dimension and scale level corresponding to the candidate anchor points.

[0068] S6. For each scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the control anchor set, and compare the similarity statistics to determine the semantic unit scoring result.

[0069] S7. Summarize the semantic unit scoring results and aggregate them by scale dimension to obtain dimension scores. Convert the dimension scores according to dimension weights and reverse scoring marks to obtain the total score, and output the dimension scores and total scores.

[0070] In this embodiment, S1 specifically includes:

[0071] S11. Receive the assessment text, which is derived from the natural language text or structured content entered by the test taker in accordance with the assessment requirements during the assessment process.

[0072] S12. Receive scale configuration data, which comes from a pre-set assessment scale file or database record.

[0073] S13. Scale dimensions represent the number of indicators used to measure the status of the test subjects in the assessment; scale levels represent the number of levels used to classify the evaluation level in the assessment; dimension weights represent the weight values ​​of each scale dimension in the comprehensive evaluation; reverse scoring markers are used to identify scale dimensions that need to be reverse-scored.

[0074] S14. Perform a standardized formatting operation on the assessment text to generate standardized assessment text.

[0075] In this embodiment, S2 specifically includes:

[0076] S21. Call the LLaMA large language model to receive the standardized assessment text, perform overall semantic parsing on the standardized assessment text, and identify the topic semantics, behavioral semantics, emotional semantics and cognitive semantics contained in the standardized assessment text;

[0077] S22. Based on the semantic parsing results, detect the semantic transfer position and semantic integrity boundary in the standardized evaluation text, and divide the standardized evaluation text into several mutually distinguishable semantic unit texts, each semantic unit text corresponding to a natural language content.

[0078] S23. Number the generated semantic unit texts to form a semantic unit text set.

[0079] In this embodiment, the semantic parsing operation specifically includes:

[0080] Before performing semantic parsing, input the normalized evaluation text into LLaMA and input the parsing instructions to limit the scope of parsing. The parsing instructions are used to constrain LLaMA to perform semantic analysis and semantic decomposition only on the normalized evaluation text.

[0081] During the semantic parsing process, LLaMA reads the standardized evaluation text segment by segment, judges the semantic consistency of consecutive sentences in the text, and records the corresponding position as the semantic splitting position when it detects that the topic content has changed, the described object has changed, or the semantic direction has changed.

[0082] During the semantic decomposition process, if there are parallel descriptions, transitional descriptions or progressive descriptions in the standardized evaluation text, LLaMA will combine the relevant statements into the same semantic unit text according to the principle of semantic integrity.

[0083] After generating semantic unit text, a unique identifier is assigned to each semantic unit text, and the original order of the semantic unit texts in the standardized evaluation text is maintained.

[0084] When LLaMA fails to identify clear semantic boundaries during semantic parsing or semantic decomposition, the corresponding continuous text is output as a semantic unit text.

[0085] The resulting collection of semantic unit texts is output as structured data.

[0086] In this embodiment, the parameters of LLaMA are configured as follows:

[0087] The LLaMA model employs an autoregressive decoding approach with a parameter size on the order of 7 bytes and a context window length of 4096 tokens. During the inference phase, the decoding temperature parameter is set to 0.2, the Top-p sampling parameter to 0.9, and the Top-k sampling parameter to 50. The maximum number of generated tokens is set to 2048, and the minimum number of generated tokens is set to 1. The repetition penalty parameter is set to 1.1. The model input consists only of normalized evaluation text and parsing instruction text. The model output is limited to structured text format.

[0088] In this embodiment, S3 specifically includes:

[0089] S31. Read the scale dimension field and scale level field from the scale configuration data, and for each scale dimension and each scale level, obtain the text description data corresponding to the scale dimension and scale level.

[0090] S32. Associate the text description data with the corresponding scale dimension identifiers and scale level identifiers one by one to generate a rating anchor record that includes scale dimension identifiers, scale level identifiers and text description content.

[0091] S33. Traverse all rating anchor records and group rating anchor records with the same scale dimension identifier and consecutive scale level identifiers into adjacent anchor record groups; the scale level identifier is a sortable identifier; the continuity determination is completed by comparing the numerical order of the scale level identifiers; when the scales of two rating anchor records are adjacent in numerical sorting order, they are determined to be continuous.

[0092] S34. Mark the rating anchor records, divide the rating anchor records with different scale dimensions into non-same dimension anchor record groups, and divide the rating anchor records with different scale levels into non-same level anchor record groups.

[0093] S35. Store the scoring anchor record, adjacent anchor record group, non-same dimension anchor record group, and non-same level anchor record group to generate a scoring anchor library.

[0094] In this embodiment, S4 specifically includes:

[0095] S41. Call the SBERT sentence vector model to perform vector encoding processing on each semantic unit text to generate semantic unit vectors corresponding to each semantic unit text.

[0096] S42. Call the SBERT sentence vector model to perform vector encoding processing on the text description content in the scoring anchor record one by one to generate anchor vectors corresponding to each scoring anchor record.

[0097] S43. Perform vector format unification processing on semantic unit vectors and anchor vectors. Vector format unification processing includes vector dimension consistency verification, vector numerical type unification, and vector normalization. Consistency verification involves reading the vector length values ​​of semantic unit vectors and anchor vectors respectively; when the vector length values ​​are equal, the dimension consistency verification is passed; when the vector length values ​​are not equal, the current vector processing flow is terminated or the vector encoding operation is re-executed; normalization involves calculating the vector norm for semantic unit vectors and anchor vectors respectively, using the method of dividing each vector component by the corresponding vector norm.

[0098] In this implementation, the SBERT parameters are configured as follows:

[0099] The SBERT model is based on the Transformer encoding structure, with a parameter scale on the order of 110M. The maximum text length of the model input is set to 512 tokens, and text exceeding the maximum length is truncated. The model output vector dimension is fixed at 768 dimensions, with semantic unit vectors and anchor vectors using the same dimension. Random sampling is not enabled during the model inference phase, and deterministic forward computation is used in the encoding process. The model inference batch size is set to 32, and dynamic batch size adjustment is not performed during the encoding process. The model weight parameters are initialized during the model loading phase and remain unchanged during the encoding of semantic unit text and scoring anchor text. Gradient calculation is not enabled during the model inference phase, and all vector encoding operations are performed in inference mode.

[0100] In this embodiment, S5 specifically includes:

[0101] S51. For each semantic unit vector, calculate the vector similarity between the semantic unit vector and all anchor vectors; the vector similarity is calculated using cosine similarity.

[0102] S52. Sort the vector similarity results corresponding to each semantic unit vector according to their numerical values, and select anchor vectors whose similarity values ​​are within a predetermined range as candidate anchors; the predetermined number is 10.

[0103] S53. Read the scale dimension identifier and scale level identifier corresponding to the candidate anchor point, and combine the scale dimension identifier and scale level identifier contained in the candidate anchor point corresponding to the same semantic unit vector to generate a set of rating hypotheses. Each rating hypothesis consists of a set of scale dimension identifiers and scale level identifiers. For each candidate anchor point, read the corresponding scale dimension identifier and scale level identifier. Generate a rating hypothesis for each candidate anchor point corresponding to the same semantic unit vector.

[0104] S54. Perform deduplication on the set of rating hypotheses, delete rating hypotheses whose scale dimension identifiers and scale level identifiers are exactly the same, retain unique rating hypotheses, and generate a set of rating hypotheses corresponding to semantic units.

[0105] In this embodiment, S5 specifically includes:

[0106] S61. For each rating hypothesis corresponding to the same semantic unit, calculate the similarity statistic between the semantic unit vector and the anchor vector in the rating anchor record corresponding to the rating hypothesis, and obtain the target similarity value of the rating hypothesis.

[0107] S62. Compare the target similarity value with the first threshold. When the target similarity value is less than the first threshold, terminate the verification process of the scoring hypothesis. The first threshold is set to 0.65.

[0108] S63. When the target similarity value is greater than or equal to the first threshold, calculate the similarity statistic between the semantic unit vector and the anchor vector in the adjacent anchor record group. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value is within the preset level tolerance range, retain the current scoring hypothesis and record the adjacent levels; the tolerance range is 0.05.

[0109] S64. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value exceeds the preset level tolerance range, calculate the similarity statistic between the semantic unit vector and the anchor vector in the non-same level anchor record group. When the similarity statistic corresponding to the non-same level anchor record group is greater than the target similarity value, terminate the current scoring hypothesis and generate a new level scoring hypothesis.

[0110] S65. When the similarity statistic corresponding to the anchor record group of different levels is less than or equal to the target similarity value, calculate the similarity statistic between the semantic unit vector and the anchor vector in the anchor record group of different dimensions.

[0111] S66. When the similarity statistic corresponding to the non-same dimension anchor point record group is greater than the target similarity value, terminate the scoring hypothesis under the current scale dimension and transfer the semantic unit to the scoring process corresponding to the non-same dimension.

[0112] S67. When the target similarity value is greater than the similarity statistics corresponding to the adjacent anchor record group, the non-same level anchor record group, and the non-same dimension anchor record group, the current scoring hypothesis is determined as the scoring hypothesis that passes the screening.

[0113] In this embodiment, the scoring process described in S66 specifically includes:

[0114] S661. For the same semantic unit, read the set of rating hypotheses that have passed the filtering, and extract the scale dimension identifier and scale level identifier corresponding to each rating hypothesis.

[0115] S662. When the set of rating hypotheses that have passed the screening is empty, the semantic unit is marked as a low-confidence semantic unit and the output of the rating result of the semantic unit is terminated.

[0116] S663. When the set of rating hypotheses that has passed the screening is not empty, determine whether the scale level identifiers in the set of rating hypotheses are the same.

[0117] S664. When all scale level identifiers in the rating hypothesis set are the same, the scale level identifier is used as the level result of the semantic unit, and the corresponding scale dimension identifier is used as the dimension result of the semantic unit.

[0118] S665. For the rating hypotheses corresponding to different scale levels in the rating hypothesis set, compare the target similarity with the target similarity of the rating hypotheses corresponding to adjacent scale levels respectively; when the target similarity of a rating hypothesis corresponding to a certain scale level is greater than the target similarity of the rating hypotheses corresponding to adjacent scale levels, select the scale level as the rating result of the semantic unit.

[0119] S666. When all rating hypotheses corresponding to the scale levels in the rating hypothesis set fail to be validated, the semantic unit is marked as a low-confidence semantic unit, and the output of the rating result of the semantic unit is terminated.

[0120] S667. Output the scale dimension identifier and scale level result of the semantic unit as the scoring result of the semantic unit.

[0121] In this embodiment, S7 specifically includes:

[0122] S71. Collect the semantic units with determined rating results according to the scale dimension identifiers, and extract the scale level results of the corresponding semantic units for each scale dimension identifier.

[0123] S72. For each scale dimension identifier, perform numerical summary processing on the collected scale level results to obtain the dimension score corresponding to the scale dimension identifier.

[0124] S73. Read the dimension weights and reverse scoring markers in the scale configuration data, perform reverse conversion and weighted calculation on the dimension scores corresponding to each scale dimension identifier, summarize to obtain the total evaluation score, and output the dimension scores corresponding to each scale dimension identifier and the total evaluation score.

[0125] Example 1:

[0126] To verify the feasibility of this invention in practice, it was applied to a comprehensive assessment scenario involving open-ended text completion. In this scenario, test takers need to describe their state, behavioral performance, and subjective feelings using natural language, focusing on several assessment dimensions. Unlike traditional assessment methods that rely solely on multiple-choice questions, this type of assessment text varies in length and expression, containing a large amount of implicit semantics and cross-sentence connections, making it difficult to process manually. Previous methods typically involved measurement personnel exporting text data from the backend after the assessment, followed by manual reading, splitting, categorizing, and scoring to complete the scoring process. This process was time-consuming and prone to misunderstandings between different measurement personnel, leading to insufficient scoring consistency.

[0127] In this embodiment, the completed assessment text is directly input into the method flow of this invention. The text is formatted uniformly to eliminate format differences, resulting in standardized text. Subsequently, a large language model is used to perform overall semantic analysis on the standardized text, splitting the continuous text into multiple semantic units according to semantic integrity. Each semantic unit corresponds to a relatively independent semantic expression, avoiding semantic breaks or merging errors caused by subjective judgment during manual splitting.

[0128] After semantic unit decomposition, the system generates scoring anchors based on the dimensions and level descriptions defined in the scale configuration, and converts both semantic units and scoring anchors into vector representations. During the scoring process, instead of directly assigning a level based on a single similarity result, multiple scoring hypotheses are first formed, and then filtered through comparison and verification using adjacent levels, different levels, and different dimensions of anchors. When multiple level candidates exist, the system executes the verification process in order of level to avoid the risk of misjudgment caused by simply selecting the highest similarity level. When all verifications fail, the corresponding semantic unit is marked as a low-confidence semantic unit and does not participate in subsequent score aggregation.

[0129] In this embodiment, a total of 1200 text samples were collected, containing 3480 semantic units. Under manual scoring, the average time required to complete the segmentation and scoring of all texts was approximately 240 man-hours. Using the method of this invention, the average time for the complete scoring process was approximately 18 man-hours, with manual intervention limited to result sampling and anomaly verification. The average consistency rate between the manual scoring results and the scoring results of this invention reached 93.4% at the dimensional level, and after excluding low-confidence semantic units, the consistency rate increased to 96.1%. With multiple participants in manual scoring, the standard deviation of dimensional scores among different scorers was approximately 0.72, while using the method of this invention, the standard deviation of dimensional scores decreased to 0.18, improving scoring stability.

[0130] In the results aggregation stage, the method of this invention can automatically collect semantic unit scoring results according to scale dimensions and generate the final evaluation score by combining dimension weights and reverse scoring tags. Compared with manual aggregation, it avoids repetitive operations and human errors in the manual conversion and statistical process. Comparative analysis shows that, under the same evaluation sample conditions, the total score distribution generated by the method of this invention is consistent with the overall trend of manual scoring results, but the dispersion of abnormally high or low scores is reduced, indicating that the invention has better stability and controllability in complex text evaluation scenarios.

[0131] Table 1: Comparison of evaluation results of different methods at the semantic level

[0132] Sample size Number of semantic units Manual scoring takes time Automatic scoring time Dimensional consistency rate Consistency rate (excluding low confidence levels) Human rating standard deviation Automatic scoring standard deviation 1200 3480 240 18 93.4% 96.1% 0.72 0.18

[0133] As shown in Table 1, under the same sample size, manually completing the text segmentation and scoring requires a significant investment of manpower and time. The method of this invention, however, reduces overall processing time, requiring only a small amount of human intervention for result verification to complete the entire scoring process. Regarding scoring consistency, the automatic evaluation method maintains a high degree of consistency with human scoring at the dimensional level. Furthermore, after removing low-confidence semantic units, the consistency rate further improves, indicating that this invention can effectively identify and avoid interference from semantically ambiguous text in the scoring results. Simultaneously, human scoring exhibits significant score dispersion when multiple participants are involved, while the scoring results generated by the method of this invention have a lower standard deviation, reflecting a more stable scoring scale, which is beneficial for horizontal comparison and vertical analysis of the assessment results. In summary, the data in this table fully validates that this invention can simultaneously improve processing efficiency and scoring quality in practical applications.

[0134] The above description is only a preferred embodiment of the present invention, but the scope of protection of the present invention is not limited thereto. Any equivalent substitutions or modifications made by those skilled in the art within the scope of the technology disclosed in the present invention, based on the technical solution and inventive concept of the present invention, should be covered within the scope of protection of the present invention.

Claims

1. An automatic evaluation method for assessment data based on a machine learning-based semantic recognition model, characterized in that, Includes the following steps: S1. Receive the assessment text and scale configuration data, perform preprocessing on the assessment text to obtain standardized assessment text, and the scale configuration data package contains scale dimensions, scale levels, dimension weights and reverse scoring tags. S2. Call LLaMA to perform semantic parsing on the standardized evaluation text, identify semantic topics and split them according to semantic boundaries to generate semantic unit text; S3. Read the scale configuration data to generate a scoring anchor library, and use the set of control anchors to organize and verify the scale dimensions and scale levels. S4. Call the SBERT sentence vector model to perform vector encoding on the semantic unit text to obtain a set of semantic unit vectors, and perform vector encoding on the anchor text of the scoring anchor library to obtain a set of anchor vectors. S5. Perform similarity retrieval on each semantic unit vector to obtain candidate anchor points, and generate a set of rating hypotheses based on the scale dimension and scale level corresponding to the candidate anchor points. S6. For each scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the scoring hypothesis, calculate the similarity statistic between the semantic unit vector and the anchor vector corresponding to the control anchor set, and compare the similarity statistics to determine the semantic unit scoring result. S7. Summarize the semantic unit scoring results and aggregate them by scale dimension to obtain dimension scores. Convert the dimension scores according to dimension weights and reverse scoring marks to obtain the total score, and output the dimension scores and total scores.

2. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 1, characterized in that, S2 specifically includes: S21. Call the LLaMA large language model to receive the standardized assessment text, perform overall semantic parsing on the standardized assessment text, and identify the topic semantics, behavioral semantics, emotional semantics and cognitive semantics contained in the standardized assessment text; S22. Based on the semantic parsing results, detect the semantic transfer position and semantic integrity boundary in the standardized evaluation text, and divide the standardized evaluation text into several mutually distinguishable semantic unit texts, each semantic unit text corresponding to a natural language content. S23. Number the generated semantic unit texts to form a semantic unit text set.

3. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 2, characterized in that, S3 specifically includes: S31. Read the scale dimension field and scale level field from the scale configuration data, and for each scale dimension and each scale level, obtain the text description data corresponding to the scale dimension and scale level. S32. Associate the text description data with the corresponding scale dimension identifiers and scale level identifiers one by one to generate a rating anchor record that includes scale dimension identifiers, scale level identifiers and text description content. S33. Traverse all rating anchor records and divide the rating anchor records with the same scale dimension identifier and consecutive scale level identifiers into adjacent anchor record groups. S34. Mark the rating anchor records, divide the rating anchor records with different scale dimensions into non-same dimension anchor record groups, and divide the rating anchor records with different scale levels into non-same level anchor record groups. S35. Store the scoring anchor record, adjacent anchor record group, non-same dimension anchor record group, and non-same level anchor record group to generate a scoring anchor library.

4. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 3, characterized in that, S4 specifically includes: S41. Call the SBERT sentence vector model to perform vector encoding processing on each semantic unit text to generate semantic unit vectors corresponding to each semantic unit text. S42. Call the SBERT sentence vector model to perform vector encoding processing on the text description content in the scoring anchor record one by one to generate anchor vectors corresponding to each scoring anchor record. S43. Perform vector format unification processing on semantic unit vectors and anchor vectors. Vector format unification processing includes vector dimension consistency verification, vector numerical type unification, and vector normalization processing.

5. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 4, characterized in that, S5 specifically includes: S51. For each semantic unit vector, calculate the vector similarity between the semantic unit vector and all anchor vectors. S52. Sort the vector similarity results corresponding to each semantic unit vector according to the numerical value, and select anchor vectors whose similarity values ​​are within a predetermined range as candidate anchors. S53. Read the scale dimension identifier and scale level identifier corresponding to the candidate anchor point, and combine the scale dimension identifier and scale level identifier contained in the candidate anchor point corresponding to the same semantic unit vector to generate a set of rating hypotheses. Each rating hypothesis consists of a set of scale dimension identifier and scale level identifier. S54. Perform deduplication on the set of rating hypotheses, delete rating hypotheses whose scale dimension identifiers and scale level identifiers are exactly the same, retain unique rating hypotheses, and generate a set of rating hypotheses corresponding to semantic units.

6. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 5, characterized in that, S5 specifically includes: S61. For each rating hypothesis corresponding to the same semantic unit, calculate the similarity statistic between the semantic unit vector and the anchor vector in the rating anchor record corresponding to the rating hypothesis, and obtain the target similarity value of the rating hypothesis. S62. Compare the target similarity value with the first threshold. When the target similarity value is less than the first threshold, terminate the verification process of the scoring hypothesis. S63. When the target similarity value is greater than or equal to the first threshold, calculate the similarity statistic between the semantic unit vector and the anchor vector in the adjacent anchor record group. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value is within the preset level tolerance range, retain the current scoring hypothesis and record the adjacent levels. S64. When the difference between the similarity statistic corresponding to the adjacent anchor record group and the target similarity value exceeds the preset level tolerance range, calculate the similarity statistic between the semantic unit vector and the anchor vector in the non-same level anchor record group. When the similarity statistic corresponding to the non-same level anchor record group is greater than the target similarity value, terminate the current scoring hypothesis and generate a new level scoring hypothesis. S65. When the similarity statistic corresponding to the anchor record group of different levels is less than or equal to the target similarity value, calculate the similarity statistic between the semantic unit vector and the anchor vector in the anchor record group of different dimensions. S66. When the similarity statistic corresponding to the non-same dimension anchor point record group is greater than the target similarity value, terminate the scoring hypothesis under the current scale dimension and transfer the semantic unit to the scoring process corresponding to the non-same dimension. S67. When the target similarity value is greater than the similarity statistics corresponding to the adjacent anchor record group, the non-same level anchor record group, and the non-same dimension anchor record group, the current scoring hypothesis is determined as the scoring hypothesis that passes the screening.

7. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 6, characterized in that, The scoring process described in S66 specifically includes: S661. For the same semantic unit, read the set of rating hypotheses that have passed the filtering, and extract the scale dimension identifier and scale level identifier corresponding to each rating hypothesis. S662. When the set of rating hypotheses that have passed the screening is empty, the semantic unit is marked as a low-confidence semantic unit and the output of the rating result of the semantic unit is terminated. S663. When the set of rating hypotheses that has passed the screening is not empty, determine whether the scale level identifiers in the set of rating hypotheses are the same. S664. When all scale level identifiers in the rating hypothesis set are the same, the scale level identifier is used as the level result of the semantic unit, and the corresponding scale dimension identifier is used as the dimension result of the semantic unit. S665. For the rating hypotheses corresponding to different scale levels in the rating hypothesis set, compare the target similarity with the target similarity of the rating hypotheses corresponding to adjacent scale levels respectively; when the target similarity of a rating hypothesis corresponding to a certain scale level is greater than the target similarity of the rating hypotheses corresponding to adjacent scale levels, select the scale level as the rating result of the semantic unit. S666. When all rating hypotheses corresponding to the scale levels in the rating hypothesis set fail to be validated, the semantic unit is marked as a low-confidence semantic unit, and the output of the rating result of the semantic unit is terminated. S667. Output the scale dimension identifier and scale level result of the semantic unit as the scoring result of the semantic unit.

8. The automatic evaluation method for assessment data based on a machine learning-based semantic recognition model according to claim 7, characterized in that, Specifically, S7 includes: S71. Collect the semantic units with determined rating results according to the scale dimension identifiers, and extract the scale level results of the corresponding semantic units for each scale dimension identifier. S72. For each scale dimension identifier, perform numerical summary processing on the collected scale level results to obtain the dimension score corresponding to the scale dimension identifier. S73. Read the dimension weights and reverse scoring markers in the scale configuration data, perform reverse conversion and weighted calculation on the dimension scores corresponding to each scale dimension identifier, summarize to obtain the total evaluation score, and output the dimension scores corresponding to each scale dimension identifier and the total evaluation score.

Images