Performance consistency analysis method, system, device, and medium

Abstract

Embodiments of the present application provide a score consistency analysis method, system, device and medium, belonging to the field of examination and evaluation. The method is based on the test score data, supports custom field mapping and question type rule configuration, automatically analyzes and aggregates fine-grained question type scores; according to the grouping strategy, the data is divided into independent subgroups, and the missing and low score noise data is removed; in each subgroup, multi-dimensional detection logic is executed: based on Z-score statistics, question type scores significantly lower than the mean are identified, and the isolated forest algorithm is applied to mine score mode outliers in high-dimensional space. Finally, combined with t-SNE dimension reduction and DBSCAN density clustering algorithm, a visual distribution map is generated to locate noise points, that is, clustering structure anomalies, that is, groups with inconsistent scores. It can effectively identify hidden abnormal score patterns, and provide intelligent technical support for score review and quality monitoring of large-scale examinations.

Description

Technical Field

[0001] This invention relates to the field of examination and assessment technology, and specifically to a method, system, device, and medium for analyzing the consistency of performance. Background Technology

[0002] In large-scale standardized testing, computer-based testing has gradually become the mainstream testing format. To adapt to differences in hardware infrastructure across regions, testing systems typically need to be deployed on various terminal devices. For example, some testing areas use dedicated desktop computers, others use tablets, and remote testing areas may only support smartphone access. Although the test questions are the same across different testing terminals for the same exam, the computer-based testing software requires custom development during the adaptation process due to differences in screen resolution, interaction methods, and operating system kernels. This multi-terminal heterogeneous deployment environment introduces potential technical risks: in the automatic scoring module, misconfiguration or other issues on specific devices may lead to systematic scoring errors or data omissions for specific question types on certain types of devices. Such errors are often subtle, only showing abnormalities in fine-grained question type scores, and may be masked by scores on other questions at the overall score level, making them difficult to detect using traditional total score verification methods.

[0003] Currently, consistency checks on exam scores primarily rely on manual review mechanisms, using threshold screening based on total scores and random sampling. However, existing traditional methods have limitations: First, there is a lack of fine-grained analysis of question types. Traditional methods rely on the total score indicator, which makes it difficult to effectively identify situations where the total score is normal but some question types are abnormal. For example, if a test area experiences a problem with the testing equipment or a data leak that causes all candidates in that area to automatically score zero for a certain question type, while these candidates score normally on other questions, their total score may only show a slight decrease, which is insufficient to trigger an anomaly alarm based on the total score, thus leading to a large-scale systemic scoring incident.

[0004] Second, it is difficult to cope with data noise caused by factors beyond the examinee's ability. In actual examination administration, in addition to equipment compatibility issues, human errors such as incorrect score registration and misconfiguration of key values ​​for objective question answers may occur. These errors can lead to distortions in the scores of specific question types for specific groups of examinees, resulting in deviations from a normal distribution. Such anomalies caused by factors beyond the examinee's ability can often only be passively discovered in the post-examination stage, lacking a proactive early warning mechanism.

[0005] Third, traditional methods are poorly adapted to population distribution. If traditional statistical methods such as Z-score are directly applied to the full dataset, they are very likely to misjudge the real low-scoring candidates as data anomalies, or mask the local anomaly patterns in the middle score range due to the existence of the high-scoring group. Summary of the Invention

[0006] The purpose of this invention is to provide a method, system, device, and medium for analyzing performance consistency, which can effectively distinguish between insufficient ability and systematic errors, thereby ensuring the fairness and data accuracy of large-scale examinations.

[0007] To achieve the above objectives, embodiments of the present invention provide a method for analyzing performance consistency, comprising: Obtain the raw exam score data, and group it according to the level field and subject field contained in the raw exam score data to generate multiple data subgroups, and remove the records with the absence mark in each data subgroup; Based on the preset question type aggregation rules, regular expressions are used to match and aggregate the scores of each sub-question belonging to the same question type, so as to obtain the question type score vector for each candidate. Within each data subgroup, multi-dimensional anomaly detection is performed on the question type score vector of each examinee to obtain statistical outlier examinees and examinees with abnormal scoring patterns. The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest. Based on the question type score vector of each candidate, a dimensionality reduction algorithm is used to reduce the dimensionality of the data to obtain two-dimensional coordinate points for each candidate. Then, a density clustering algorithm is used to cluster the two-dimensional coordinate points. Points that cannot be classified into normal clusters are marked as noise points, and a visual scatter plot is generated. Candidates with noise points are highlighted in the plot. The noise points represent candidates with abnormal clustering structures.

[0008] Optionally, the preset question type aggregation rule is represented as: a user-defined question prefix string for each question type and the corresponding question number range.

[0009] Optional, Z-score-based statistical outlier detection includes: Calculate the total score distribution of all candidates within the current data subgroup and set a threshold for excluding low-scoring candidates; Determine the critical value of the total score based on the ranking of total scores; For any candidate, if their total score is less than the total score threshold, the candidate's question type anomaly judgment is skipped; if their total score is greater than the total score threshold, Z-score statistics are calculated for their scores on multiple question types. If the Z-score statistic for any question type is less than a preset threshold, the candidate is marked as a statistical outlier.

[0010] Optional pattern recognition detection based on isolation forests includes: An isolated forest model consisting of multiple isolated trees is constructed, and the contamination rate parameter of the isolated forest model is adjusted according to the sample size in the current data subgroup. A dynamic judgment threshold is determined based on the contamination rate parameter. Normalize the score vectors of each question type for each candidate in the current data subgroup to obtain the normalized sample; The normalized sample is input into each isolated tree of the isolated forest model, and the number of edges from the root node to the leaf node of the isolated tree is calculated to obtain the path length. Calculate the arithmetic mean of the path lengths of the sample across all isolated trees, and define it as the expected value; Based on the expected value and the theoretical expected value, the abnormal scores of each candidate in the current data subgroup are obtained; If a candidate's abnormal score is greater than or equal to the dynamic judgment threshold, the candidate is judged to have an abnormal scoring pattern; otherwise, the candidate is judged to be normal.

[0011] Optionally, anomaly scores can be calculated using the following formula:

[0012] In the formula, For the expected value, This represents the theoretical expected value of the average path length for a given sample size n.

[0013] Optionally, based on the question type score vector of each candidate, a dimensionality reduction algorithm is used to reduce its dimensionality, obtaining two-dimensional coordinates for each candidate, including: The t-SNE algorithm is used to map the question type score vector in the high-dimensional space to a two-dimensional space, and this is achieved by minimizing the KL divergence of the conditional probability distribution between the high-dimensional and low-dimensional spaces.

[0014] In the formula, This indicates that in higher-dimensional space, the examinee Centered on the candidates The probability of being chosen as their neighbor; In a two-dimensional mapping space, a point As a point The probability of being neighbors.

[0015] Optionally, after obtaining the original exam score data and grouping it according to the level and subject fields contained in the original exam score data, the score consistency analysis method further includes: minimum sample threshold determination. The system counts the number of candidates within each group based on the level and subject fields, and determines whether the number of candidates in each group is less than the preset minimum sample threshold. If the number of candidates in a certain group is less than the preset minimum sample threshold, the anomaly detection process for that group will be skipped.

[0016] Secondly, the present invention also provides a performance consistency analysis system, comprising: The data acquisition module is used to acquire raw exam score data, group it according to the level field and subject field contained in the raw exam score data, generate multiple data subgroups, and remove records with absence flags in each data subgroup; The feature engineering module is used to match and aggregate the scores of each sub-question belonging to the same question type based on preset question type aggregation rules, and obtain the question type score vector for each candidate. The dual-dimensional detection module is used to perform multi-dimensional anomaly detection on the question type score vector of each candidate in each data subgroup to obtain statistical outlier candidates and candidates with abnormal scoring patterns. The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest. The visualization analysis module is used to reduce the dimensionality of each candidate's question type score vector using a dimensionality reduction algorithm to obtain two-dimensional coordinate points for each candidate. Then, it uses a density clustering algorithm to cluster the two-dimensional coordinate points, marking points that cannot be classified into normal clusters as noise points, generating a visualization scatter plot, and highlighting the candidates with noise points in the plot. The noise points represent candidates with abnormal clustering structures.

[0017] Thirdly, the present invention also provides an electronic device, including a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor executes the program to implement the steps of the above-described performance consistency analysis method.

[0018] Fourthly, the present invention also provides a storage medium storing a computer program thereon, which, when executed by a processor, implements the steps of the above-described performance consistency analysis method.

[0019] By employing the above technical solutions, the original score data is grouped and cleaned; a fine-grained question type feature matrix is ​​constructed using regularization rules; low-score noise interference is removed using Z-score statistics combined with a total score lower limit filtering mechanism; outliers in the scoring patterns in high-dimensional space are identified using the isolated forest algorithm; and a visual distribution map is generated by combining t-SNE dimensionality reduction and DBSCAN density clustering. This enables the accurate location and differentiation of score anomalies and systematic scoring errors for specific question types. It solves the technical problem that traditional total score verification methods are unable to detect local anomalies, reduces false alarm rates, improves detection efficiency, and ensures the fairness and data accuracy of large-scale examinations.

[0020] Other features and advantages of the embodiments of the present invention will be described in detail in the following detailed description section. Attached Figure Description

[0021] The accompanying drawings are provided to further illustrate embodiments of the present invention and form part of the specification. They are used together with the following detailed description to explain the embodiments of the present invention, but do not constitute a limitation thereof. In the drawings: Figure 1 This is a flowchart of a performance consistency analysis method provided in an embodiment of the present invention; Figure 2 This is a schematic diagram illustrating the principle of dynamic parameter adjustment and scoring for isolated forests provided in an embodiment of the present invention; Figure 3 This is a visual analysis diagram provided by an embodiment of the present invention; Figure 4 This is a schematic diagram of the structure of a performance consistency analysis system provided in an embodiment of the present invention. Detailed Implementation

[0022] Various embodiments of this disclosure will be described more fully in the following detailed description. This disclosure may have various embodiments, and adjustments and changes may be made therein. However, it should be understood that there is no intention to limit the various embodiments of this disclosure to the specific embodiments disclosed herein, but rather this disclosure should be understood to cover all adjustments, equivalents, and / or alternatives falling within the spirit and scope of the various embodiments of this disclosure.

[0023] In the following, the terms “comprising” or “may include”, which may be used in various embodiments of this disclosure, indicate the presence of the disclosed functions or operations and do not limit the addition of one or more functions or operations. Furthermore, as used in various embodiments of this disclosure, the terms “comprising,” “having,” and their cognates are intended only to indicate a specific feature, number, step, operation, or combination of the foregoing and should not be construed as primarily excluding the presence of one or more other features, numbers, steps, operations, or combinations of the foregoing, or the possibility of adding one or more features, numbers, steps, operations, or combinations of the foregoing.

[0024] In various embodiments of this disclosure, the expression "or" or "at least one of A and / or B" includes any combination or all combinations of the words listed simultaneously. For example, the expression "A or B" or "at least one of A and / or B" may include A, may include B, or may include both A and B.

[0025] The technical solutions of the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings. Obviously, the described embodiments are only some embodiments of the present invention, and not all embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those skilled in the art without creative effort are within the scope of protection of the present invention.

[0026] See Figure 1 The diagram shows a flowchart of a performance consistency analysis method in a specific embodiment, including the following execution steps: S100: Obtain the original exam score data, and group it according to the level field and subject field contained in the original exam score data to generate multiple data subgroups, and remove the records with the absence mark in each data subgroup.

[0027] Specifically, after executing step S100, the following steps are also performed: The system counts the number of examinees in each group based on the level and subject fields, and determines whether the number of examinees in each group is less than the preset minimum sample threshold. If the number of examinees in a group is less than the preset minimum sample threshold, the anomaly detection process for that group is skipped.

[0028] More specifically, the system acquires the original exam score data file, loads and parses the data file through a graphical user interface, provides a field mapping configuration interface allowing users to map column names in the data file to predefined key fields, which at least include a candidate identifier field, a level field, a subject field, a total score field, and an absence flag field; and divides the entire dataset based on the level and subject fields to generate multiple independent data subgroups. ,in Indicates the first Grouping; statistical analysis of each data subgroup. Number of valid candidates ,like Less than the preset minimum sample threshold If so, skip the subsequent detection process for that group; in each data subgroup Within the dataset, records with empty absence flags or non-specific absence codes are filtered out, and the absence data is removed to obtain a clean candidate dataset.

[0029] In one specific embodiment, the system provides a file loading function through a graphical user interface. After the user uploads the exam score file, the system automatically parses the file structure. Subsequently, the system enters the field mapping configuration stage. Since the field names of exported files vary between different exam systems, this embodiment provides a flexible mapping mechanism. Users can map the original column names to standard fields one by one through drop-down selections. For example, the "Admission Ticket Number" column in the original file is mapped to "Candidate Identifier," and the "Absent" column is mapped to "Absent Flag." After the mapping is completed, the system loads the original data into the memory data structure and performs preliminary format validation to ensure that key fields have no null values ​​and that the data types are correct. An example of desensitized data is shown in Table 1 below. The content is for illustrative purposes only and has no actual meaning.

[0030] Table 1: Sample of anonymized data in grade files

[0031] In one specific embodiment, after the data is loaded, the system performs grouping and cleaning operations: First, the data is grouped, and the system divides the entire dataset into multiple independent subgroups based on the user-configured level and subject fields. Each subgroup represents a group of candidates with the same exam level and subject. This grouping strategy ensures that consistency testing is conducted within homogeneous groups, avoiding interference between candidates from different levels and subjects.

[0032] Then, a minimum sample threshold is determined, and the number of valid test takers in each subgroup is counted. If a certain group If the sample size is less than the preset minimum sample threshold, the system will skip the subsequent detection process for that group to avoid misjudgments caused by the failure of the statistical model under small sample sizes.

[0033] Next, absentee data was removed. Within each subgroup, records with an empty absence flag or a specific non-absence code were selected. Records marked as absent were directly removed and not included in subsequent analysis.

[0034] Finally, low-score noise filtering is performed. The system calculates the total score distribution of all candidates within the current subgroup, determines the total score ranking, and sets a preset percentage threshold. In subsequent Z-score detection, only candidates with total scores above this threshold are evaluated for question type scores. If a candidate's total score is below the threshold, even if their Z-score for a particular question type is low, they are not marked as an anomaly, thus effectively distinguishing between insufficient ability and data anomalies.

[0035] S101: Based on the preset question type aggregation rules, regular expressions are used to match and aggregate the scores of each sub-question belonging to the same question type to obtain the question type score vector for each candidate.

[0036] Specifically, the preset question type aggregation rule is represented as: the user-defined question prefix string for each question type and the corresponding question number range.

[0037] In one specific implementation, a question type rule configuration interface is provided, allowing users to input question prefixes and number ranges. Based on the rules, the system automatically matches data columns using regular expressions, aggregates and generates columns for each question type, and constructs a fine-grained question type feature matrix. ,in, Indicates the first The candidate in Scores for each question type.

[0038] For example, to enable fine-grained analysis of exam scores, this embodiment uses question type rule configuration, allowing users to define question type aggregation rules based on the actual exam paper structure. For example: Multiple choice questions: The prefix is ​​"T", and the number range is 1-50; Multiple choice questions: those prefixed with "T" and numbered in the range of 51-70; Fill in the blanks: The prefix is ​​"T", and the number range is 71-85; Comprehensive question: The prefix is ​​"T", and the number range is 86-100.

[0039] The system automatically generates regular expressions based on user-configured rules, iterates through all column names in the original data, and matches the score columns that conform to the rules. For example, the score columns for single-choice questions are T1-T50, and the score columns for multiple-choice questions are T51-T70, etc., precisely matching the column names in the original data table.

[0040] For each matched question type, the system aggregates the scores of all sub-questions belonging to that question type, i.e., performs a summation operation, to generate the total score column for that question type.

[0041] S102: Within each data subgroup, perform multi-dimensional anomaly detection on the question type score vector of each candidate to obtain statistical outlier candidates and candidates with abnormal scoring patterns.

[0042] The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest.

[0043] Specifically, statistical outlier detection based on Z-score includes the following sub-steps: S1: Calculate the total score distribution of all candidates in the current data subgroup and set a threshold for excluding low-scoring candidates.

[0044] S2: Determine the critical value of the total score based on the total score ranking.

[0045] Specifically, calculate the total score distribution of all candidates within the current group and set a threshold for excluding candidates with low scores. (Usually 0.20), the total score threshold is determined based on the total score ranking. :

[0046] in, This is the ascending order of the total scores of the candidates within this group. This indicates rounding down to the nearest integer.

[0047] S3: For any candidate, if their total score is less than the total score threshold, skip the question type anomaly judgment for that candidate; if their total score is greater than the total score threshold, calculate the Z-score statistic for their multiple question type scores.

[0048] Specifically, for any candidate If its total score < If so, skip the question type anomaly check for that candidate; if > Then for its first Score for each question type Calculate the Z-score statistic :

[0049] in, For the first in this group The average score for each question type. The standard deviation is calculated using the following formulas: ,

[0050] S4: If the Z-score statistic for any question type is less than the preset threshold, then mark the candidate as a statistical outlier.

[0051] If any question type exists Make < ( If a preset threshold (usually -2.0) is used, the candidate is marked as a statistical outlier, meaning that the candidate... In terms of question types The score was significantly lower than that of the candidate's group of candidates.

[0052] Specifically, pattern recognition detection based on isolated forests includes the following sub-steps: SA: Construct an isolated forest model consisting of multiple isolated trees, adjust the pollution rate parameter of the isolated forest model according to the sample size in the current data subgroup, and determine the dynamic judgment threshold based on the pollution rate parameter.

[0053] For details, see Figure 2 As shown, a forest model consisting of n_trees isolated trees is constructed, where n_trees represents the total number of trees in the forest and determines the stability and convergence of the model. The more trees there are, the more stable the expected path length, but the computation time also increases. Considering that the number of examinees in large-scale exams usually exceeds 5000, to ensure accurate detection and control computational resources, this embodiment fixes n_trees at 100. The contamination rate parameter is used to predict the proportion of abnormal samples in the dataset and directly affects the threshold cut-off point for consistency determination. This embodiment is based on the sample size of the current group. The value of contamination is dynamically adjusted. The specific rules are as follows: When sample size With a sample size ≥5000, based on the law of large numbers, the proportion of outliers tends to stabilize. Therefore, a contamination rate parameter of 0.01 is set to rigorously screen for extreme outliers. Within the interval [100, 5000), considering the large statistical fluctuations with small samples, the threshold is appropriately relaxed, and the contamination rate parameter is set to 0.03 to prevent missed reports due to sample sparsity. If the value is less than 100, this logic will not be executed to avoid statistical distortion in small samples.

[0054] SB: Normalize the score vectors for each question type for each candidate in the current data subgroup to obtain the normalized sample.

[0055] Specifically, the score vectors for each question type for each candidate within the current group are calculated. After normalization, the features are mapped to the [0,1] interval and used as input features for the isolated forest model.

[0056] in, and Each of the following is the first in this group. The maximum and minimum scores for each question type, and the normalized vector. As input features for isolated forests, the influence of dimensions is eliminated, ensuring that each question type has consistent weights in anomaly detection.

[0057] SC: Input the normalized sample into each isolated tree of the isolated forest model, calculate the number of edges from the root node to the leaf node of the isolated tree, and obtain the path length.

[0058] Specifically, for each normalized sample This is then input into each isolation tree. Path length Defined as a sample The number of edges traversed from the root node to a leaf node in a tree. In an isolated forest, outliers are usually few and short, making them easier to isolate by random splits, resulting in shorter path lengths in the tree; normal nodes, on the other hand, require more splits to be isolated, resulting in longer path lengths.

[0059] SD: Calculate the arithmetic mean of the path lengths of the sample across all isolated trees and define it as the expected value.

[0060] Because a forest contains multiple trees, the path length of a sample within each tree may differ. Expected value This represents the arithmetic mean of the path lengths of the sample across all n_trees trees:

[0061] in, This represents the path length of the sample in the k-th tree.

[0062] SE: Based on the expected value and the theoretical expected value, obtain the abnormal score of each candidate in the current data subgroup.

[0063] Specifically, the anomaly score is calculated using the following formula:

[0064] In the formula, For the expected value, Given sample size The theoretical expected value of the average path length (usually 256) is used to... Normalization is performed to make it unaffected by the total sample size. The formula is:

[0065] in For harmonic series.

[0066] SF: If a candidate's abnormal score is greater than or equal to the dynamic judgment threshold, the candidate is judged to have an abnormal scoring pattern; otherwise, the candidate is judged to be normal.

[0067] Specifically, based on the dynamically adjusted contamination rate parameter mentioned above, the system executes the following judgment logic: First, it calculates the set of abnormal scores for all candidates in the current group. And sorted from largest to smallest; based on the dynamically set contamination value (denoted as...). Determine the dynamic judgment threshold , Defined as a score set The Middle The percentile score; if a candidate's score is abnormal. If the score is normal, the candidate is considered to have an abnormal scoring pattern; otherwise, the candidate is considered to have a normal scoring pattern.

[0068] S103: Based on the question type score vector of each candidate, a dimensionality reduction algorithm is used to reduce the dimensionality of the vector to obtain the two-dimensional coordinate points of each candidate. The density clustering algorithm is then used to cluster the two-dimensional coordinate points. Points that cannot be classified into normal clusters are marked as noise points, and a visual scatter plot is generated. Candidates with noise points are highlighted in the plot.

[0069] The noise points represent candidates with abnormal cluster structures.

[0070] Specifically, based on each candidate's question type score vector, a dimensionality reduction algorithm is used to reduce its dimensionality, resulting in two-dimensional coordinates for each candidate, including: Extract the question type feature matrix within the current group, and use the t-SNE algorithm to extract the high-dimensional question type features. Mapping to a two-dimensional space yields the two-dimensional coordinates of each examinee. Its optimization objective is to minimize the KL divergence between the conditional probability distributions in the high-dimensional space and the low-dimensional space:

[0071] In the formula, This indicates that in higher-dimensional space, the examinee Centered on the candidates The probability of being chosen as their neighbor; In a two-dimensional mapping space, a point As a point The probability of being neighbors.

[0072] The density-based DBSCAN algorithm was applied to perform density clustering on two-dimensional coordinate points to identify dense groups of test takers and discrete outliers. Based on the DBSCAN clustering results, the data points were divided into three categories: Core and boundary points: These belong to a dense cluster, representing a typical group of test-takers with a consistent scoring pattern. Noise points: Points that cannot be assigned to any cluster, marked as having anomalies in cluster structure. These test-takers typically exhibit unusual question-type score combinations, belonging neither to the high-scoring nor low-scoring groups, possibly due to specific answering strategies or system misjudgment. A two-dimensional scatter plot is generated, rendering test-takers in different clusters with different colors based on DBSCAN's cluster labels. Test-takers with abnormal cluster structures are highlighted (in red). See [link / reference]. Figure 3 As shown.

[0073] In one specific implementation, the list of candidates with abnormal scores marked in the first logic (statistical outlier detection based on Z-score), the second logic (pattern recognition detection based on isolated forest), and the cluster analysis is summarized and deduplicated; an anomaly type label is generated for each candidate, such as Z-score anomaly or isolated forest anomaly, and the structured data containing candidate identifier, name, total score, abnormal question type, and anomaly type is exported as an Excel report file.

[0074] For each candidate with an anomaly, an anomaly type label is generated, such as Z-score anomaly or isolated forest anomaly. The structured data, including candidate identifier, name, total score, anomaly question type, and anomaly type, is then exported as an Excel report file. An example of the anonymized report is shown in Table 2 below; this is for illustrative purposes only and does not contain any actual information.

[0075] Table 2: Examples of anonymized data in report documents

[0076] In this embodiment, based on student performance data, it supports custom field mapping and question type rule configuration, automatically parses and aggregates fine-grained question type scores; divides the data into independent subgroups according to a grouping strategy, and removes noisy data such as those of absentees and low scores; performs multi-dimensional detection logic within each subgroup: identifies question type scores significantly below the mean based on Z-score statistics, and applies the isolated forest algorithm to mine outliers in the scoring patterns in high-dimensional space; finally, combines t-SNE dimensionality reduction and DBSCAN density clustering algorithm to generate a visual distribution map to locate noise point groups, thereby detecting abnormal scores and ensuring the fairness and data accuracy of large-scale examinations.

[0077] It should be understood that the sequence number of each step in the above embodiments does not imply the order of execution. The execution order of each process should be determined by its function and internal logic, and should not constitute any limitation on the implementation process of the embodiments of the present invention.

[0078] like Figure 4 As shown, the following are embodiments of the performance consistency analysis system provided in this disclosure. The performance consistency analysis methods in the above embodiments belong to the same inventive concept. For details not described in detail in the embodiments of the performance consistency analysis system, please refer to the embodiments of the above performance consistency analysis methods.

[0079] A performance consistency analysis system, comprising: The data acquisition module is used to acquire raw exam score data, group it according to the level field and subject field contained in the raw exam score data, generate multiple data subgroups, and remove records with absence flags in each data subgroup; The feature engineering module is used to match and aggregate the scores of each sub-question belonging to the same question type based on preset question type aggregation rules, and obtain the question type score vector for each candidate. The dual-dimensional detection module is used to perform multi-dimensional anomaly detection on the question type score vector of each candidate in each data subgroup to obtain statistical outlier candidates and candidates with abnormal scoring patterns. The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest. The visualization analysis module is used to reduce the dimensionality of each candidate's question type score vector using a dimensionality reduction algorithm to obtain two-dimensional coordinate points for each candidate. Then, it uses a density clustering algorithm to cluster the two-dimensional coordinate points, marking points that cannot be classified into normal clusters as noise points, generating a visualization scatter plot, and highlighting the candidates with noise points in the plot. The noise points represent candidates with abnormal clustering structures.

Claims

1. A method for analyzing the consistency of academic performance, characterized in that, include: Obtain the raw exam score data, and group it according to the level field and subject field contained in the raw exam score data to generate multiple data subgroups, and remove the records with the absence mark in each data subgroup; Based on the preset question type aggregation rules, regular expressions are used to match and aggregate the scores of each sub-question belonging to the same question type, so as to obtain the question type score vector for each candidate. Within each data subgroup, multi-dimensional anomaly detection is performed on the question type score vector of each examinee to obtain statistical outlier examinees and examinees with abnormal scoring patterns. The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest. Based on the question type score vector of each candidate, a dimensionality reduction algorithm is used to reduce the dimensionality of the data to obtain two-dimensional coordinate points for each candidate. Then, a density clustering algorithm is used to cluster the two-dimensional coordinate points. Points that cannot be classified into normal clusters are marked as noise points, and a visual scatter plot is generated. The noise points and candidates marked as abnormal are highlighted in the plot. The noise points represent candidates with abnormal clustering structures.

2. The performance consistency analysis method according to claim 1, characterized in that, The preset question type aggregation rule is represented as: the user-defined question prefix string for each question type and the corresponding question number range.

3. The performance consistency analysis method according to claim 1, characterized in that, Z-score-based statistical outlier detection includes: Calculate the total score distribution of all candidates within the current data subgroup and set a threshold for excluding low-scoring candidates; Determine the critical value of the total score based on the ranking of total scores; For any candidate, if their total score is less than the total score threshold, the candidate's question type anomaly judgment is skipped; if their total score is greater than the total score threshold, Z-score statistics are calculated for their scores on multiple question types. If the Z-score statistic for any question type is less than a preset threshold, the candidate is marked as a statistical outlier.

4. The performance consistency analysis method according to claim 1, characterized in that, Pattern recognition and detection based on isolated forests include: An isolated forest model consisting of multiple isolated trees is constructed, and the contamination rate parameter of the isolated forest model is adjusted according to the sample size in the current data subgroup. A dynamic judgment threshold is determined based on the contamination rate parameter. Normalize the score vectors of each question type for each candidate in the current data subgroup to obtain the normalized sample; The normalized sample is input into each isolated tree of the isolated forest model, and the number of edges from the root node to the leaf node of the isolated tree is calculated to obtain the path length. Calculate the arithmetic mean of the path lengths of the sample across all isolated trees, and define it as the expected value; Based on the expected value and the theoretical expected value, the abnormal scores of each candidate in the current data subgroup are obtained; If a candidate's abnormal score is greater than or equal to the dynamic judgment threshold, the candidate is judged to have an abnormal scoring pattern; otherwise, the candidate is judged to be normal.

5. The performance consistency analysis method according to claim 4, characterized in that, Calculate the anomaly score using the following formula: In the formula, For the expected value, This represents the theoretical expected value of the average path length for a given sample size n.

6. The performance consistency analysis method according to claim 1, characterized in that, Based on each candidate's question type score vector, a dimensionality reduction algorithm is used to reduce its dimensionality, resulting in two-dimensional coordinates for each candidate, including: The t-SNE algorithm is used to map the question type score vector in the high-dimensional space to a two-dimensional space, and this is achieved by minimizing the KL divergence of the conditional probability distribution between the high-dimensional and low-dimensional spaces. In the formula, This indicates that in higher-dimensional space, the examinee Centered on the candidates The probability of being chosen as their neighbor; In a two-dimensional mapping space, a point As a point The probability of being neighbors.

7. The performance consistency analysis method according to claim 1, characterized in that, After obtaining the original exam score data and grouping it according to the level and subject fields contained in the original exam score data, the score consistency analysis method further includes: minimum sample threshold determination. The system counts the number of candidates within each group based on the level and subject fields, and determines whether the number of candidates in each group is less than the preset minimum sample threshold. If the number of candidates in a certain group is less than the preset minimum sample threshold, the anomaly detection process for that group will be skipped.

8. A performance consistency analysis system, characterized in that, include: The data acquisition module is used to acquire raw exam score data, group it according to the level field and subject field contained in the raw exam score data, generate multiple data subgroups, and remove records with absence flags in each data subgroup; The feature engineering module is used to match and aggregate the scores of each sub-question belonging to the same question type based on preset question type aggregation rules, and obtain the question type score vector for each candidate. The dual-dimensional detection module is used to perform multi-dimensional anomaly detection on the question type score vector of each candidate in each data subgroup to obtain statistical outlier candidates and candidates with abnormal scoring patterns. The multi-dimensional anomaly detection includes statistical outlier detection based on Z-score and pattern recognition detection based on isolated forest. The visualization analysis module is used to reduce the dimensionality of each candidate's question type score vector using a dimensionality reduction algorithm to obtain two-dimensional coordinate points for each candidate. Then, it uses a density clustering algorithm to cluster the two-dimensional coordinate points, marking points that cannot be classified into normal clusters as noise points, generating a visualization scatter plot, and highlighting the candidates with noise points in the plot. The noise points represent candidates with abnormal clustering structures.

9. An electronic device comprising a memory, a processor, and a computer program stored in the memory and executable on the processor, characterized in that, When the processor executes the program, it implements the steps of the performance consistency analysis method as described in any one of claims 1 to 7.

10. A storage medium having a computer program stored thereon, characterized in that, When the computer program is executed by the processor, it implements the steps of the performance consistency analysis method as described in any one of claims 1 to 7.

Images